Saturday, February 6, 2016

Improved Documentation of HotSpot Options in Java 8

One of the small but welcome features introduced with Oracle's HotSpot implementation of Java 8 is the addition of many common HotSpot Java launcher (java) options/flags to the launcher's documentation. In the past, a developer interested in even some fairly common HotSpot JVM options has had to potentially look in multiple different sources to find "official" documentation on these flags. These sources included the aforementioned Java launcher documentation as we all other resources such as Troubleshooting Guide for Java SE 6 with HotSpot VM, Troubleshooting Guide for HotSpot VM (7), and Java 8 Troubleshooting Guide. There have also been numerous "unofficial" sources documenting these options such as in books and The most complete list of -XX options for Java JVM.

One of the risks to be aware of when reading about HotSpot JVM options from disparate sources is that it is easy to start reading about a certain JVM option not realizing that the documentation is explicitly for JVMs other than HotSpot such as JRockit or IBM's JVM. JRockit's documentation can be particularly misleading if the developer is not paying close attention because it is available with an Oracle URL and with Oracle logos.

An example of JRockit JVM option documentation being confused as HotSpot JVM option documentation is a feedback comment on the blog post How to Fix java.lang.OufOfMemoryError: Direct Buffer Memory that "corrects" the author's statement about the default JVM maximum direct buffer limit by referencing JRockit documentation (but JRockit's default of 0 leads to very different behavior than HotSpot's default also of 0). In a different but related example, a poster on the StackOverflow thread Default for XX:MaxDirectMemorySize makes the same mistake and also references the JRockit documentation despite the question being, "What is the default value for XX:MaxDirectMemorySize for SUN JVM 1.6?" Having common HotSpot options documented in the main Java launcher documentation should help reduce these frequent causes of confusion about HotSpot VM options' behavior.

Two examples of HotSpot JVM options now documented directly in the Java launcher (the executable java to which the options apply) documentation are -XX:+HeapDumpOnOutOfMemory and -XX:MaxDirectMemorySize. The documentation for the Oracle HotSpot 8 Java launcher describes these two example options as shown next:

These and several other HotSpot JVM options that were available before Java 8 are not described in prior versions of the Java launcher documentation such as for Java 7 and Java 6. This is a small but nonetheless welcome addition that is just one small part of a long list of reasons for moving to Java 8 from older versions.

Monday, January 25, 2016

Book Review: JavaScript Concurrency

Adam Boduch's book JavaScript Concurrency has been published by Packt Publishing (December 2015). The subtitle of the book is, "Build better software with concurrent JavaScript programming, and unlock a more efficient and forward thinking approach to web development." JavaScript Concurrency consists of ten chapters spanning over 260 substantive pages. The author has described this book in his blog post of the same title.

Preface

The Preface of JavaScript Concurrency provides a single sentence summary of each of the book's ten chapters (find longer chapter descriptions in the author's blog post). The Preface states that the "requirements for this book" are a modern web browser, Node.js, and a code editor.

JavaScript Concurrency's Preface states that "all aspects of concurrent, asynchronous, and parallel programming are covered from first principles" and advertises that the book is "written for any JavaScript developer who wants to learn how to write more efficient, powerful, and maintainable applications that utilize the latest developments in the JavaScript language."

Chapter 1: Why JavaScript Concurrency?

The first sentence of the first chapter of JavaScript Concurrency opens with, "JavaScript is not a language associated with concurrency," but then points out that "this has changed a lot over the past few years, especially with new language features in ES 2015. The chapter introduces concepts of synchronous and asynchronous JavaScript. There is coverage of asynchronous versus parallel that uses simple juggler examples to illustrate the differences.

JavaScript Concurrency's initial chapter introduces the "JavaScript concurrency principles" of "parallelize," "synchronize," and "conserve." The chapter describes the "parallelize principle" as "taking advantage of modern CPU capabilities to compute results in less time" and explains that JavaScript parallelism is achieved in web browsers with Web Workers and in Node.js by spawning processes.

Chapter 1 describes the "synchronize principle" as "about the mechanisms used to coordinate concurrent actions and the abstractions of those mechanisms" and introduces callback functions and Promises. The chapter describes the "conserve principle" as "about saving on compute and memory resources" and introduces "lazy evaluation."

Chapter 2: The JavaScript Execution Model

JavaScript Concurrency's second chapter introduces the JavaScript execution model and related terminology: tasks, execution environment, interpreter, task queue, and event loop. The section "Creating tasks using timers" introduces and demonstrates use of setTimeout() and setInterval() "to schedule JavaScript code to run at a later time."

The second chapter of JavaScript Concurrency looks at basic ways of "responding to DOM events" and "responding to network events" using JavaScript callbacks. As part of this discussion, discussion and demonstrative code listings are provided related to use of debounce and for coordinating multiple requests. The chapter concludes with discussion of some of the challenges of using JavaScript callbacks as an introduction to the motivation for promises.

Chapter 3: Synchronizing with Promises

The third chapter of JavaScript Concurrency opens with a brief history of promises in JavaScript and defines key terms associated with promises: Promise, State, Executor, Resolver, Rejector, and Thenable. The objective of this chapter is to cover how the ES6 implementation of promises in JavaScript helps address JavaScript's so-called "callback hell."

Chapter 3 provides explanations of each of the terms related to JavaScript promises that were briefly introduced at the beginning of the chapter. The chapter uses multiple code listings and descriptive text to explain different approaches to using promises and I liked the id it presents of extending the promise prototype with an always() function that should be run for a given promise whether it resolves or rejects.

JavaScript Concurrency's third chapter reinforces that promises "are born into a pending state, and they die in either a resolved or rejected state" and then goes onto describe consequence of this. There is more in-depth coverage of use of JavaScript promises in different contexts that includes discussion and example code illustrating Promise.all() for handling multiple promises, Promise.race() for specifying that first resolved promise wins, and Promise.resolve(), and Promise.reject() for dealing with a JavaScript function that has "the potential to be both synchronous and asynchronous."

Chapter 4: Lazy Evaluation with Generators

Chapter 4 of JavaScript Concurrency opens with the sentence, "Lazy evaluation is a programming technique, which is used when we don't want to compute values until the very last second." The early portion of this chapter also explains, "The Generator is a new primitive type introduced to JavaScript as a part of the ES6 specification of the language" that helps "implement lazy evaluation techniques in [JavaScript] code." The chapter looks at the JavaScript Generator in more detail, including coverage of generator function syntax (function*), use of yield and next, and use of for .. of for iterating over a sequence of generators..

JavaScript Concurrency's fourth chapter describes with text and illustrates with code how to use generators to work with infinite and alternating/circular sequences. The chapter similarly describes and illustrates deferring between generators and interweaving generators. There are also sections in this chapter regarding passing data to generators, reusing general generators, and implementing "lightweight map/reduce." The "Coroutines" section of Chapter 4 "looks at some techniques for implementing coroutines in JavaScript using generators" and provides examples of implementing and using JavaScript coroutines based on generators.

Chapter 5: Working with Workers

The fifth chapter of JavaScript Concurrency begins with the sentence, "Web workers enable true parallelism within a web browser," and states, "In this chapter, we'll walk through the conceptual ideas of web workers, and how they relate to the concurrency principles that we're trying to achieve in our applications." The chapter introduces web workers as "at their core, ... nothing more than operating system-level threads."

Chapter 5 covers three types of web workers: dedicated (default), sub-workers, and shared workers. There is also coverage of worker environments and how these differ from the main thread's environment.

JavaScript Concurrency's fifth chapter explains and demonstrates the worker importScripts() function to import the lodash library. The chapter also covers communication between workers and related message serialization. I like that the chapter includes discussion of trade-offs and disadvantages associated with use of shared workers and sub-workers. The chapter concludes with coverage of error and exception handling in web workers.

Chapter 6: Practical Parallelism

Chapter 6 of JavaScript Concurrency introduces functional programming and related concepts such as immutability and referential transparency. A section in this chapter asks, "Do we need to go parallel?" That section briefly discusses how to decide whether to apply parallelism or not. This is complemented by a section that briefly discusses how to determine how many cores are available and use that information to decide how much parallelism is appropriate. The chapter also introduces "candidate problems" that can be addressed with parallelism such as mapping and reducing, searching collections, and "keeping the DOM responsive."

Chapter 7: Abstracting Concurrency

The seventh chapter begins by looking "at the approaches that we might use to insulate the rest of our application from tricky concurrency bits." The chapter explains how promises can be used to encapsulate the complexities of concurrent code. The explanations and code examples demonstrate "creating helper functions that wrap the worker communications" to return promises and "extending the web worker interface" postMessage() method. The chapter's section on "lazy workers" looks at "using generators in web workers."

Several of the code listings in Chapter 7 are at least partially taken from http://adambom.github.io/parallel.js/ (and this is designated as code comments in those listings). After referencing these multiple code listings based on that library, the author formally introduces Parallel.js and explains that this library's purpose is "to make interacting with web workers as seamless as possible." The chapter discusses how to use Parallel.js to provide one's own functions to workers provided by Parallel.js rather than implementing one's own workers. The chapter demonstrates using Parallel.js to spawn workers and for mapping and reducing. I like that this section includes discussion of parallel slowdown and how to address that.

Chapter 7 concludes with coverage of worker pools. There are several pages devoted to explanation and demonstrative code listings related to implementing a worker "pool abstraction [that] look[s] and behave[s] like a plain dedicated worker."

Chapter 8: Evented IO with NodeJS

The purpose of Chapter 8 is to "explain how Node handles concurrency and how we need to program our NodeJS code to take full advantage of this environment." The chapter introduces evented IO, the Node.js Event Loop, and why these work well in web environments. The author introduces Dan Kegel's concept of the C10K problem and discusses how Node helps address this. The areas of specific focus are network IO and file system IO with descriptive text explanations and associated code samples.

Chapter 9: Advanced NodeJS Concurrency

Chapter 9 begins by introducing the co library as a library relying on generators and promises "to implement coroutines." The introduction to co introduces the "co() function to create a coroutine," demonstrates how co's awaiting values support is similar to proposed ES7 syntax, demonstrates resolving promises with co, and explains using "the wrap() utility to make a plain coroutine function into a reusable function."

JavaScript Concurrency's ninth chapter includes a section that explains how to fork new Node child processes to avoid issues with the blocked IO event loop. There is also a section on applying "a pool of general-purpose processes." The "Server Clusters" section discusses "scaling our Node application to several machines" and covers examples of this in the context of proxying, microservices, and load balancing.

Chapter 10: Building a Concurrent Application

The final chapter of JavaScript Concurrency uses a "simple chat application" to demonstrate concepts covered in the earlier chapters for building a concurrent application in JavaScript. The chapter begins by revisiting why it's easier to design applications to be concurrent from the beginning than to retrofit an application that was not designed to be concurrent.

The sample chat application built in Chapter 10 is based on Node.js and uses a few libraries such as commander and formidable. Although several simplifying assumptions are made to keep the example simpler, there is still a lot of code to list and describe. The chapter concludes with a section on other features that could be added to the sample chat application and with a paragraph that closes out the entire book.

General Observations

  • JavaScript Concurrency covers several modern (ES6/ES7) topics. This is very useful as many JavaScript books don't yet cover these modern abilities. However, the one drawback is that some of the covered material is not finalized and is subject to change. This is a trade-off that every book written about "cutting edge" topics faces.
  • Several JavaScript libraries with concurrent-orientation are introduced and demonstrated in JavaScript Concurrency. The most attention is paid to Parallel.js, co, formidable, and commander, probably in that order.
  • JavaScript Concurrency not only introduces syntax of new JavaScript concurrency features, but it also includes discussion on motivations for using the different constructs and trade-offs and design decisions that should be considered when selecting a JavaScript concurrent construct.
  • JavaScript Concurrency is best suited for an intermediate or advanced JavaScript developer who has little or no experience with promises, generators, workers, coroutines, and other ES6/ES7 concurrency concepts. A potential reader of this book should probably understand the material covered in Concurrency Model and Event Loop before reading this book. I wouldn't recommend this book to someone trying to learn JavaScript with no previous experience with the language, but would recommend it to someone with basic familiarity and some experience with JavaScript.
  • JavaScript Concurrency covers JavaScript concurrency both in web browser environments and on the server with Node.js.
  • There are numerous code listings in JavaScript Concurrency. These code listings are black text on white background with no color syntax and no line numbers. The code is available for download, which is probably preferable so that the code can be viewed in a favorite editor or IDE with color syntax and line numbers. The code listings are heavily commented (more so and at a lower level than I'd want in production code) and this is desirable in a book that is introducing and explaining concepts. It's nice to be able to read the code and associated comments to gain understanding and switch less between the code and associated text discussion for that understanding.
  • JavaScript Concurrency contains numerous simple diagrams to illustrate concepts. These diagrams consist of simple shapes such as boxes and arrows. In some cases, these simple diagrams make a concept easier to understand and in some cases the diagrams add very little value perhaps because they are overly simplistic.
  • There are some typos in JavaScript Concurrency that generally did not preclude me from understanding the points being made. I list a sample of them here so that readers of this post can decide if these are significant to their understanding, mildly distracting, or not an issue at all.
    • Page 100 has text introducing a code snippet that states, "Next, we'll use the loadScripts() function to bring the lodash library into our library...", but fortunately the actual code listing correctly demonstrates importScripts() rather than loadScripts().
    • Page 149, "Our application is likely filled with operations these, where concurrency simply doesn't make sense."
    • Page 233, "it will be a very simply chat application"
  • Because preferences can be subjective and opinionated, I like to include references to other reviews of the same books in many of my book reviews. Other reviews of JavaScript Concurrency are available:

Conclusion

JavaScript Concurrency sticks to what its two-words title describes: JavaScript concurrency. It is in many ways a "modern JavaScript" book that added clarity to my understanding of what JavaScript now supports or soon will support in many different browsers and on the server with Node.js.

Monday, January 4, 2016

Leaner Java Collections with FastUtil

In response to my recent post Discovering a Trove of Java Primitives Collection Handling on the GNU Trove library, TheAlchemist pointed out some advantages of fastutil over trove: "I much prefer fastutil (http://fastutil.di.unimi.it/), because it's still in active development, has more features, supports large sizes (> 2^32), and has better documentation." Attila-Mihaly Balazs has seconded this: "I second @TheAlchemist's recommendation for fastutil! It's a great library." In this post, I look at fastutil from some of the same perspectives that I previously looked at trove.

The main fastutil page describes fastutil as an extension of the JavaTM Collections Framework that provides "type-specific maps, sets, lists and queues with a small memory footprint and fast access and insertion" along with "big (64-bit) arrays, sets and lists, and fast, practical I/O classes for binary and text files." The license for fastutil is Apache License, Version 2 and the current version of fastutil requires Java 7 or newer. There are currently (as of this writing) "unmaintained" versions of fastutil available for download as well for Java 6 and Java 5 as well.

Adding elements to a FastUtil collection is accomplished with the same API calls as used with standard Java collections as demonstrated in the next code listing which compares inserting elements into a JDK ArrayList to inserting elements into a FastUtil DoubleArrayList.

Inserting Doubles with JDK and Inserting doubles with FastUtil DoubleArrayList
/**
 * Demonstrate standard JDK {@code ArrayList<Double>}
 * with some JDK 8 functionality.
 */
public void demonstrateJdkArrayListForDoubles()
{
   final ArrayList<Double> doubles = new ArrayList<>();
   doubles.add(15.5);
   doubles.add(24.4);
   doubles.add(36.3);
   doubles.add(67.6);
   doubles.add(10.0);
   out.println("JDK ArrayList<Double>:");
   out.println("\tDoubles List: " + doubles);
}

/**
 * Demonstrate use of DoubleArrayList and show how
 * similar using it is to using {@code ArrayList<Double>}.
 */
public void demonstrateFastUtilArrayListForDoubles()
{
   // Demonstrate adding elements to DoubleArrayList is
   // exactly like adding elements to ArrayList<Double>.
   final DoubleArrayList doubles = new DoubleArrayList();
   doubles.add(15.5);
   doubles.add(24.4);
   doubles.add(36.3);
   doubles.add(67.6);
   doubles.add(10.0);
   out.println("FastUtil DoubleArrayList:");  // DoubleArrayList overrides toString()
   out.println("\tDoubles List: " + doubles);
}

When the two above methods are executed, the list of doubles that are written to standard output appear exactly the same with even the same square braces surrounding the comma-separated doubles values.

FastUtil collections tend to implement the appropriate JDK collection interfaces. For example, the just-demonstrated class DoubleArrayList implements several interfaces including Collection<Double> and List<Double>. It turns out that DoubleArrayList also implements it.unimi.dsi.fastutil.doubles.DoubleStack and it.unimi.dsi.fastutil.Stack<Double>. The ability to use this class as a stack is demonstrated in the next code listing.

Using FastUtil's DoubleArrayList as a Stack
/**
 * Demonstrate FastUtil's Double Stack.
 *
 * FastUtil's DoubleStack allows access to its
 * contents via push, pop, and peek. It is declared
 * as a DoubleArrayList type here so that the size()
 * method is available without casting.
 */
public void demonstrateFastUtilDoubleStack()
{
   final DoubleArrayList stack = new DoubleArrayList();
   stack.push(15.5);
   stack.push(17.3);
   stack.push(16.6);
   stack.push(2.2);
   out.println("FastUtil Stack of Doubles");
   out.println("\tPeek: " + stack.peek(0) + "; After Size: " + stack.size());
   out.println("\tPop:  " + stack.pop() + "; After Size: " + stack.size());
   out.println("\tPeek: " + stack.peek(0) + "; After Size: " + stack.size());
}

As I discussed in the blog post on Trove, Trove provides a gnu.trove.TCollections class that is an analogous (subset) to java.util.Collections. FastUtil provides similar functionality, but this approach of providing static methods to act upon FastUtil collections is separated into type-specific and structure-specific classes with static methods rather than in a single class with static methods. The next code listing demonstrates using one of these type-specific and structure-specific classes with static methods, IntSets, in conjunction with a FastUtil IntLinkedOpenHashSet. As the name suggests, the IntSets class provides "static methods and objects that do useful things with [int]-specific sets."

Using IntSets with IntLinkedOpenHashSet
/**
 * Demonstrate one of FastUtil's "equivalent"s of the
 * java.util.Collections class. FastUtil separates its
 * grouping of static methods into classes that are
 * specific to the data type of the collection and to
 * the data structure type of the collection.
 */
public void demonstrateFastUtilCollectionsClass()
{
   final IntLinkedOpenHashSet integers = new IntLinkedOpenHashSet();
   integers.add(5);
   integers.add(7);
   integers.add(3);
   integers.add(1);
   final IntSet unmodifiableIntegers = IntSets.unmodifiable(integers);
   out.println("Unmodifiable Integers:");
   out.println("\tClass: " + unmodifiableIntegers.getClass().getCanonicalName());
   try
   {
      unmodifiableIntegers.add(15);
   }
   catch (Exception ex)
   {
      out.println("\tException caught: " + ex);
   }
}

FastUtil supports that standard Java iteration approaches of using an explicit iterator and using the Java SE 5-introduced for-each loop. FastUtil collections even support the JDK 8 style using .forEach (assuming code is built and run on JDK 8) because the FastUtil collections implement java.lang.Iterable. These are demonstrated in the next code listing.

Iterating FastUtil Collections in Standard Java Style
/**
 * Demonstrate "traditional" Java iteration of a
 * FastUtil collection.
 */
public void demonstrateIterationWithIterator()
{
   final LongOpenHashSet longs = new LongOpenHashSet();
   longs.add(15);
   longs.add(6);
   longs.add(12);
   longs.add(13);
   longs.add(2);
   final LongIterator longIterator = longs.iterator();
   while (longIterator.hasNext())
   {
      final long longValue = longIterator.next();
      out.print(longValue + " ");
   }
}

/**
 * Demonstrate iteration of a FastUtil collection
 * using Java's enhanced for-each approach.
 */
public void demonstrateIterationWithForEach()
{
   final LongLinkedOpenHashSet longs = new LongLinkedOpenHashSet();
   longs.add(15);
   longs.add(6);
   longs.add(12);
   longs.add(13);
   longs.add(2);
   for (final long longValue : longs)
   {
      out.println(longValue + " ");
   }
}

/**
 * Demonstrate iteration of a FastUtil collection
 * using JDK 8 .forEach approach.
 */
public void demonstrateIterationWithJdk8ForEach()
{
   final LongLinkedOpenHashSet longs = new LongLinkedOpenHashSet();
   longs.add(15);
   longs.add(6);
   longs.add(12);
   longs.add(13);
   longs.add(2);
   longs.forEach(longValue -> out.print(longValue + " "));
}
Additional Observations Related to FastUtil
  • Because FastUtil collections implement standard JDK 8 collection interfaces, the APIs are easy to pick up and use with familiar Java idioms.
  • FastUtil collections generally provide a constructor that accepts an array of the underlying data type and an overridden toArray() method and a type-specific method such as toDoubleArray() [for double-oriented collections] to provide their data elements in form of array of primitives.
  • FastUtil collections generally provide explicitly overridden toString() implementations that allow for the individual data elements to be easily written similar to JDK collections and differently than Java arrays (which require Arrays.toString() methods).
  • FastUtil's Java packages are organized generally by primitive type with specific implementations of the various data structure types for that primitive type all in the same package. For example, packages are named like it.unimi.dsi.fastutil.doubles, it.unimi.dsi.fastutil.ints, and so on.
  • Because each FastUtil collection is specific to a particular primitive data type, each collection does not require a generic parameter and has none of the issues related to generics (such as erasure). I haven't seen FastUtil take advantage of the collections being type-specific for type-specific methods like Trove does, probably because FastUtil more closely implements the corresponding Java collection interfaces.
  • FastUtil's Javadoc API documentation is probably the best place to start with when learning to use FastUtil. The classes, interfaces, enums, and packages tend to be fairly well documented in FastUtil's Javadoc-based API documentation.
Conclusion

Like Trove, FastUtil is a library that can potentially be used to more efficiently (in terms of memory and performance) work with Java collections. While Trove seems to have formerly been the most popular of the many choices available, FastUtil is perhaps the most popular currently for reasons that include those cited by TheAlchemist: "still in active development, has more features, supports large sizes (> 2^32), and has better documentation." Similar libraries besides Trove and FastUtil include High Performance Primitive Collections for Java (HPPC), Koloboke, Goldman Sachs Collections, Mahout collections, and Javolution.

Thursday, December 31, 2015

Significant Software Development Developments of 2015

This post is my personal and opinionated assessment of some of the most significant developers related to software development in 2015. My previous years' assessment are available for 2014, 2013, 2012, 2011, 2010, 2009, 2008, and 2007. As with these previous years' assessments, this assessment of 2015's major developments in software development are obviously biased, opinionated, and limited to my perspective.

Image courtesy of Stuart Miles at FreeDigitalPhotos.net
Image courtesy of Stuart Miles at FreeDigitalPhotos.net

10. Internet of Things

The concept of Internet of Things has been popular for multiple years, but it seemed like I saw it all over the non-development mainstream press more than ever in 2015. Examples of Internet of Things stories from non-development outlets include Forbes's Roundup Of Internet of Things Forecasts and Market Estimates, 2015 and 17 'Internet Of Things' Facts Everyone Should Read (their 2014 article A Simple Explanation Of 'The Internet Of Things' is also useful), The Motley Fool's The Best Internet of Things Stocks of 2015, Politico's What Washington really knows about the Internet of Things, Business Insider's How the 'Internet of Things' will affect the world, Inc.'s The 'Internet of Things' Is a Bad Idea, Fox News's Why the Internet of Things is a double-edged sword, CNN's Your TV may be watching you, ABC News's How to Keep Your Fridge From Exposing Your Data, CBS News's FTC: 'Internet of Things' poses consumer risks, NBC News's Can the 'Internet of Things' Preserve Privacy?: Lawmakers, The New York Times's Diving Headfirst Into the Internet of Things and The Internet of Way Too Many Things, and The Denver Post's Denver tech community ponders morality of building Internet of Things.

CIO.com has collected its top ten Internet of Things stories from 2015 in Top 10 Internet of Things stories of 2015.

Most of the stories cited above focus on the benefits versus the risks and trade-offs (primarily to security and privacy) of moving to the Internet of Things.

9. Cloud Computing

Cloud computing is as strong as ever. In fact, it's so strong and so prevalent, that I seriously considered leaving it off this list as such a general term just like I leave off other well-established and well-used general concepts such as Web Development, Mobile Development, REST, and Agile. However, I decided to put cloud computing on this list for at least one more year. Rick Blaisdell has published a post on his blog called 2015 Cloud Computing Recap: What has changed in the biggest vendors offerings? In this post, Blaisdell writes of ways that vendors contributed to adoption of cloud computing in 2015 related to Internet of Things, Microsoft Office 365, video delivery by IBM (and acquired Clearleap), and mobile development.

8. Single Page Application / "Native Web App"

Web developers have been working for years to move toward emulation of desktop applications in a web browser (and now on mobile devices). One could argue that the push to single-page applications (SPA) really started in earnest with the adoption of the principles that were termed Ajax clear back in 2005. In the post Native Web Apps, Henrik Joreteg articulates why he thinks the term "native web apps" (as coined by Adam Brault) is a better term for these types of web applications that are now built on "native web technologies" HTML, CSS, and JavaScript.

7. Developing Economics of Open Source

The open source software movement has been successful for many years. However, in recent years and particularly in 2015, we seem to continue to adapt to new realities of open source that we learn as open source becomes increasingly pervasive. In 2015, Pivotal dumped Groovy because it is not considered significant to their future and Groovy is now under the umbrella of the Apache Software Foundation. I found it insightful that the author of Igaro App (a JavaScript Single Page Application framework) has responded to a request to release Igaro App via Smart Open Source with this seeming truism: "The market is saturated, it doesn't matter how good your framework is, gaining traction is extremely difficult. Charging would make it impossible." Although Andrew Charnley is specifically speaking of JavaScript frameworks, I think this observation is true of many new open source software projects.

Other interesting perspectives on open source in 2015 include Why the open source business model is a failure and Bootstrapping a Business Around Open Source. It was also announced in 2015 that RoboVM would no longer be open source after its 1.8 version because the creators of RoboVM had "competitors actively exploiting our good faith by using our open source code to compete with us directly in commercial products" and because "we have received almost no meaningful contributions to our open source code."

The aforementioned Smart Open Source Movement is described as "an effort to stop the greed of corporations who use open source codes and make billions out of it while contributing nothing back to the community." The assertion is that "software developers and programmers" are ("unintentionally") "contributing to that greed of the corporate world." The Smart Open Source License attempts to allow software to which it applies to only be used "free of charge" by "any person ... [or] company NOT listed under NYSE, NASDAQ, DJIA, LSE, DAX, Shanghai Stock Exchange, Tokyo Stock Exchange, Hong Kong Stock Exchange and/or any company listed under Forbes 2000 Global list." Although I understand the motivation of this movement, I see several obstacles to it being adopted and being successful. There are concepts in the license that are completely orthogonal to the concepts of "free speech" espoused by the Free Software Foundation. It also seems difficult to apply which companies are not allowed to use the software. Is the "end user" (client) company or companies not allowed to use the software or is it the development firm that is not allowed to use the software? For example, would a NYSE-listed company not be allowed to develop a product with the framework even if all their customers were not listed on any of the exchanges? Finally, most open source contributors are at least partially motivated by recognition and the pride of seeing their work used by as many people as possible.

We also saw the end of Codehaus and the end of Google Code hosting in 2015, replaced largely by GitHub. It seems like many (perhaps most) developers use open source but a much smaller percentage contributes to open source.

6. Recognition that Polygot Programming is Not a Silver Bullet

Despite the fact that certain languages fit better in certain circumstances and despite the fact that no language works best in all situations, developers cannot help but appreciate being able to use a single programming language they know well in very varied environments. Although polygot programming offers several benefits, there are potentially significant costs associated with all affected developers needing to learn to read, write, and maintain multiple languages. This is one of the explanations for many Java developers preferring to use Java throughout an enterprise application from client ("thick client" with Swing/JavaFX, Google Web Toolkit, Vaadin, DukeScript, JSweet, JavaServer Faces, etc.) through back-end (application server/Spring and JPA implementation).

JavaScript has long been considered a client technology but the success of Node.js on the server has shown that many JavaScript developers would prefer to use a language they are already familiar with on the server rather than having to learn a new language, even when the language that is new to them exists, is mature, and fits the context well.

I believe there are advantages associated with using new languages and learning from them and their different approaches, but I can also see that the cost to an enterprise can be great if a large number of programmers are required to learn and use a wide number of languages. Therefore, it's not surprising that the enthusiasm for polygot programming seems to have reached a peak and is settling some now.

5. ECMAScript 2015 (6th Edition)

The specification behind JavaScript, ECMAScript has seen major improvements and changes in recent years. In 2015 (June), ECMA-262 6th Edition, The ECMAScript 2015 Language Specification was finalized. Many browsers already support many features of ES6 and some popular implementations of ECMAScript (such as TypeScript 1.4 and TypeScript 1.5) already support ES6 features. ECMAScript 6 has borrowed from other JavaScript-oriented frameworks and some have started to state that they will stop using CoffeeScript and return to ECMAScript 6-based JavaScript.
ECMAScript 6 specifies several new features including classes, arrow functions, constants, much-needed block scoping, promises, new data structures (maps and sets), new built-in methods, and more.

4. DevOps
The DevOps concept continued to be a much discussed topic in software development in 2015. I really like the post "How I Learned To Tune Out The DevOps Buzz: Don't believe everything you hear" because blog author Chad Schorr (a self-described "DevOps evangelist") articulates some of the concerns I've had regarding devops in practice: too much ambiguity and too many vendor-driven (self-serving) presentations on the concept. Also, my experience has proven that it's often prudent to be skeptical of anything that its evangelists argue has all positives with no costs or downsides.

In many ways, DevOps seems to be going through the same lifecycle as Agile. The question is whether DevOps will enjoy the long-term relevance that Agile has had despite suffering from some of the same issues DevOps now faces. DevOps is likely to have positive long-term repercussions on how we write and deploy software, but I also anticipate that its rapid rise in adoption will be accompanied by a rapid rise in negative or disappointing experiences with it. In the end, the idea of development and operations being aligned more closely seems like an obviously positive thing, but we will probably realize that what works for one organization doesn't always work the same for another organization and will have to learn how to implement DevOps differently in different cases. We definitely still need to get past what Mirco Hering articulates: the Dunning-Kruger effect currently frequently associated with DevOps.

Karthi Sadasivan has posted Top 5 Reasons Why Devops Will Transition Into Mainstream Strategy 2016 in which Sadasivan provides five reasons for agreeing with Gartner's expectations for DevOps to go mainstream in 2016.

3. Docker

It is next to impossible to bring up the front page of a general software development site such as DZone without seeing multiple blog and article titles mentioning Docker. In the article What is Docker and why is it so darn popular?, Steven J. Vaughan-Nichols writes that Docker "is hotter than hot because it makes it possible to get far more apps running on the same old servers and it also makes it very easy to package and ship programs." It's easy to explain the popularity of Docker, especially during the current emphasis on devops, given its numerous deployment-related benefits.

Clear back in January of 2015, Doug Dineley asked a good question, "But have we ever seen anything like Docker?"

2. Microservices

The topic of microservices continues to dominate online software development blogs and articles in 2015. Microservices, defined by James Lewis and Martin Fowler as "an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms," are touted for several benefits including ability to modularize services for different teams to implement, ability to use different technologies in different services, and the ability to make pieces of functionality less dependent on each other for deployment and execution. However, microservices, like all architectural styles, involve trade-offs and have costs. One of the most popular discussion points related to microservices is discussing how they differ from Service-Oriented Architecture (SOA). This is not all that surprising given that both architectural styles do share several characteristics and both SOA and microservices have enjoyed significant hype in the industry.

1. Software Security and Software Outages

I would like to change things up a bit and have software security issues and dysfunction not be a top development in software development for the year (#1 in 2014, #2 in 2013 with technical dysfunction as #1, and #2 in 2012), but the frequent and significant losses that are associated with software prevent me from lowering this on my list.
Issues related to software security and outages in 2015 include felony charges for eighth grade computer hacker, the hacking and exposing of data from Ashley Madison, hacking of Experian data related to 15 million people (including T-Mobile customers), U.S. presidential campaign intrigue, the rapid rise of ransom ware, more Java and Flash vulnerabilities, the Apache Commons exposed Java deserializability issue, and the hacking of data of over 20 million people in the Office of Personnel Management.

A good way to quickly realize how much of a problem hacking has been in recent years, refer to The New York Times's How Many Times Has Your Personal Information Been Exposed to Hackers? This interactive site allows the user to select organizations he or she has had data associated with in recent years to get an idea how many times his or her data was potentially hacked. Craig Lowell's post Sorting Through the Wreckage of Last Week’s Outages calls 2015 "the year of the outage."

Honorable Mention

There are always more noteworthy developments in software development than can be captured in a top ten list. This "honorable mention" section highlights other major developments that occurred in 2015 but just missed my top ten list.

Big Data

The interest in Big Data remained strong and perhaps got much stronger in 2015. Bernard Marr provides a month-by-month summary of highs and lows of Big Data in Big Data: 12 Amazing Highs And Lows Of 2015. Microsoft obviously believes Big Data is a big deal as it acquired Revolution Analytics ("a major commercial distributor of the R statistical programming language") in 2015.

Thor Olavsrud has collected CIO.com's "10 top big data and analytics stories of 2015." Tim Spann's Learning Big Data Tools in 2016 provides a "few tools for Big Data" that Spann recommends

GitHub

As mentioned earlier in my discussion on economics of open source, GitHub has already essentially strong-armed the Codehaus and Google Code public repositories out of existence by sheer rapid rise in its popularity and use by developers. Code is not the only thing being made available on GitHub; books, design documents, and various other documents and artifacts are also being "open sourced" on GitHub. There is even a software developer recruiting post titled "How to: Use GitHub to Find Super-Talented Developers" and Yegor Bugayenko has gone as far as stating, "...if your GitHub profile is not full of projects and commits, your "value" as a software developer is low..." (see the blog post for full context).

Node Foundation

The Node.js Foundation was created in 2015 as a collaboration involving the merging of Node.js and forked (2014) io.js code bases and managing future work in a "neutral" forum. The Node.js Foundation is hosted by the Linux Foundation.

OpenJDK

Java already has its own community process and guidance structure. Oracle led OpenJDK until 19 June 2015 when RedHat assumed leadership of OpenJDK. Oracle has also proposed OpenJDK: Mobile in 2015.

Emil Protalinski has written that "Google confirms next Android version will use Oracle’s open-source OpenJDK for Java APIs." Protalinski adds, "Google is replacing its implementation of the Java application programming interfaces (APIs) in Android with OpenJDK, the open source version of Oracle’s Java Development Kit (JDK). ... Google confirmed to VentureBeat that Android N will rely on an OpenJDK implementation, rather Android’s own implementation of the Java APIs."

In Analysis: Google Moving to OpenJDK, What That Really Means, Shai Almog provides some insightful analysis of the announcement that Android is replacing Harmony libraries with OpenJDK libraries and writes, "This is amazing news and a huge step forward for Java." Abel Avram writes in Android Will Use the OpenJDK that "Android makes extensive use of the Java language and several libraries, the latter being based on the retired Apache Harmony project" (Apache Harmony was retired to the Apache Attic in late 2011).

Mitch Pronschinske also cited the breaking news and has written, "Google has already contributed to the OpenJDK in the past, and they plan to make more contributions in the future as they transition fully to OpenJDK within Android. The effect of this change on the Java community could be significant, since many more developers will be familiar with Android’s libraries. The OpenJDK is much more familiar to the wider Java community than Google’s Harmony-based libraries."

Other 2015 Java-Related Developments

2015 was the year in which Java saw its developers adopting Java 8 with its lambda expressions and streams. As these approaches became more commonplace, we began to see more evidence of performance and other implications of using lambda expressions and streams. Java 9 and its flagship feature (Java modularization with Project Jigsaw) were postponed until 2017 and there was significant consternation associated with Oracle allegedly eliminating or at least trimming its Java evangelism team.

The "TIOBE Index for January 2016" (www.tiobe.com) page's "January Headline" is "Java is TIOBE's Programming Language of 2015!" The Tiobe Index for January 2016 goes onto state, "Java is currently number one in the enterprise back-end market and number one in the still growing mobile application development market (Android). Moreover, Java has become a language that integrates modern language features such as lambda expressions and streams. The future looks bright for Java." This isn't too shabby for a "dead language".

See Java in 2015 – Major happenings for more details on what happened in Java in 2015. Alex Zhitnitsky has also provided perspective on significant trends in the Java-sphere in the post "If You’ve Written Java Code in 2015 – Here Are the Trends You Couldn’t Have Missed."

Angular 2

Angular.js has enjoyed a rapid rise in popularity and usage in the web and mobile development worlds. Therefore, it's hardly surprising that Angular 2 is earning significant coverage and even controversy as its release approaches. The beta release of Angular 2 occurred in mid-December of 2015.

TypeScript

TypeScript enjoyed a highly successful 2015. In this single year, TypeScript saw the release of TypeScript 1.4 (January), TypeScript 1.5 Alpha (March), TypeScript 1.5 Beta (April), TypeScript 1.5 (huge release in July), TypeScript 1.6 Beta (September, with support for React/JSX), TypeScript 1.6 (September), TypeScript 1.6 (September), TypeScript 1.7 (November), and TypeScript 1.7.5 (December). TypeScript 1.5 included features specifically designed with Angular 2 in mind and it was announced that Angular 2 is built on TypeScript. It also appears that this means the end of AtScript.

Scala.js

It was announced in 2015 that "Scala.js is no longer experimental" as of version 0.6.0. Sébastien Doeraene also writes, "In the same way that Scala is fully interoperable with Java, Scala.js is fully interoperable with JavaScript. Scala.js can interoperate with JavaScript either in a statically or dynamically-typed way." As of this writing, Scala.js's most current version is 0.6.5 and it is "fully EcmaScript5 compatible."

Kotlin

Kotlin 1.0 Beta, Kotlin 1.0 Beta 2, Kotlin 1.0 Beta 3, and Kotlin 1.0 Beta 4 were released in 2015. The Kotlin Eclipse Plugin 0.5.0 was also released in 2015. Kotlin is described on its homepage as, "Statically typed programming language for the JVM, Android and the browser." Joseph Hager writes that "Kotlin is leading the pack of new JVM languages in the mobile space these days."

Ceylon

Ceylon 1.2.0 was released in 2015. This release introduced new features and addressed well over 1500 issues. In the post "Why you might want to choose Ceylon", Gavin King states, "Ceylon is a language that is equally at home on the JVM and on any JavaScript VM."

Groovy

Groovy was dropped by Pivotal in early 2015, but has since joined the Apache Software Foundation. The final funded Pivotal versions were Groovy 2.4 and Grails 3.0. Groovy 2.4 provides Android support.

Clojure

See Stuart Sierra's Clojure 2015 Year in Review for a summary of 2015 for Clojure and see ClojureScript Year In Review for a summary of 2015 events related to ClojureScript.

C# and .NET

C# 6.0 was released in 2015 and "ships with Visual Studio 2015". C# 6.0 includes several new features that Mark Michaelis writes will "change the way you write C# code in specific scenarios." C# 6.0 was released in conjunction with .NET Framework 4.6, F# 4, Visual Basic 14, and Visual Studio 2015.

Swift

Swift 2.0 was announced and, perhaps even more significantly, was open sourced. The site Swift.org contains more details on open sourced Swift. Swift-related source code can be found in Apple's GitHub repository.

Ruby and Ruby on Rails

In the article The state of Ruby and Rails: Opportunities and obstacles, Serdar Yegulalp writes about the current state of Ruby and Ruby on Rails with mention of the positives and the challenges. Yegulalp has also written Ruby on Rails takes on Node.js with WebSocket support, API mode where he highlights the recent release of Rails 5.0.0 beta 1. Rails 5 will only work on Ruby 2.2.2 (also released in 2015) or later so that the newest version of Rails can take advantage of Ruby 2.2 features and performance improvements and incorporate the Ruby OpenSSL Hostname Verification fix. Ruby 2.3.0 has since been released in 2015.

Perl

Perl 6 has been in the making for 15 years and 2015 is when it has finally arrived. More specifically, the beta version of the Rakudo implementation of Perl 6 was released in November and . In the Perl 6 Advent Calendar post "Christmas is here.", Coke announces the "Christmas release (December 2015) of Rakudo Perl 6 #94 'коледа'" and adds that "Rakudo is an implementation of Perl 6 on the Moar Virtual Machine."

The post 2015.52: Closer and closer to the Release provides a sense of urgency at which development of the Perl 6 specification is being conducted for a "Christmas 2015" release. Paul Krill's Developers can unwrap Perl 6 on Christmas quotes a recent Larry Wall e-mail message, "Christmas is still the plan [for release], though of course in some cultures that lasts till January 6 or so. We're just trying to nail down as many loose ends as possible before release." The Krill article also briefly looks at how Perl 6 shares some concepts with Perl 5, but is also fundamentally different (and incompatible) in other ways with Perl 5.

Python

Image courtesy of tor00722 at FreeDigitalPhotos.net

From my perspective the "other" language that has seemingly the greatest impact on Perl's future is Python. Python continues to enjoy a long-running surge in popularity. In 2015, Python experienced several new versions including Python 3.4.3 (February), Python 3.5.0a3 (March), Python 2.7.10 and Python 2.7.10 RC 1 (May), Python 3.5.0b2 (June), Python 3.5.0b3 (July), Python 3.5.0 RC 1 (August), Python 3.5.0 RC 3 (September), Python 3.5.0 RC 4 (September), Python 3.5.0 (September), Python 2.7.11 RC 1 (November), Python 2.7.11 (December), and Python 3.5.1 and Python 3.4.4 RC 1 (December).

U.S. Web Design Standards

The U.S. Web Design Standards (Alpha) were released in the summer of 2015. The purpose of this site is advertised as "Open source UI components and visual style guide to create consistency and beautiful user experiences across U.S. federal government websites." The site includes a Visual Style Guide, User Interface Components Standards, and examples of the principle and standards espoused on the site.

Spring

Josh Long's post This Year in Spring - December 29, 2015 outlines significant events in 2015 in the Spring/Pivotal world. These include Spring Cloud 1.0.0, Spring Cloud Data Flow, Spring Boot 1.3.0, and other cloud-related products. It was also announced that Spring Framework 5.0 will be delayed because "Spring 5 is designed to track JDK 9 very closely" and therefore cannot occur until after JDK 9 (which was delayed) is available.

Spring IO Platform 1.1.5 was released toward the end of 2015. This version incorporates the updated versions of many of its constituent parts including Spring Framework 4.1.9.RELEASE and Spring Boot 1.2.8.RELEASE.

Resurgence of the "Browser Wars"

With the web browser being a ubiquitous deployment environment common to desktops, laptops, and mobile devices, it's not too surprising that competition in the browser space appears to be heating up again. Jack Wallen has written in "Mozilla jettisons everything but the browser" that "Mozilla has officially announced it will cease work on Firefox OS and wants to split off Thunderbird so they can focus on one thing and one thing only. Firefox." Microsoft released Microsoft Edge as a replacement for the much maligned Internet Explorer with Windows 10 installations. The Vivaldi browser, created by developers associated with Opera, looks promising and is currently in beta.

Browsers compete on a wide variety of features and characteristics including performance, security, supported media types, and supported platforms. It will be interesting to see how renewed competition continues to help browsers move forward in all of these areas. Speaking of browsers and security, even with browsers locking down plugins (NPAPI) and favoring native HTML5, security issues still persist with plugins.

A choice of a favorite web browser is obviously a matter of taste and is subject to what a given person finds most important in a web browser. With all this stated, I found Mark Hachman's summary of how "five modern browsers" (Chrome, Opera, Firefox, Edge, and Internet Explorer) were compared to be interesting.

JSON API

JavaScript Object Notation (JSON) in an increasingly popular choice for data format used by JavaScript-based and REST-based applications. Organizations that expose their APIs via REST need to provide clients with a mechanism for understanding how to read and act upon the URIs provided with an HATEOAS approach. JSON API is one approach for "building APIs in JSON" that provides conventions that can be used when constructing JSON-based responses. In mid-2015, Dan Gebhardt announced, "After two years, four release candidates, hundreds of pull requests and issues, and countless hours of discussion, the JSON API specification has finally reached 1.0" (JSON API 1.0).

Progressive Web Apps

A concept that seems to be rapidly gaining popularity in late 2015 is Progressive Web Apps. Some good introductory resources related to this include Progressive Web Apps: Escaping Tabs Without Losing Our Soul, What Progressive Web Apps Mean for the Web, and Progressive Web Apps: ready for primetime.

Bazel

Google's Bazel build tool is advertised as, "Correct, reproducible, fast builds for everyone" and is currently in beta. Paul Gross has written the blog post Migrating from Gradle to Bazel and Sergio De Simone has written Bazel: Google Build Tool is now Open Source. It's also interesting to read the Gradle Team Perspective on Bazel (alpha version).

Conclusion

2015 was another year full of major developments in software development and this post covers some of these major events at a very high level. Although I am publishing this post on the final day of 2015, I may add some other items, details, or links as I think of them or run across them in the next several days or even weeks to form a more complete picture of the major events in software development in 2015. 2015 was another great year for software developers and, if we're not careful, the availability of programming languages, tools, frameworks, and documentation of successful practices could lead us to developing our own cases of the software developer version "affluenza."