A Brief History of SBT

Any developer that wrote more than a few lines of Scala has eventually heard of SBT. SBT is an efficient build tool with lofty functional principles, an uncommon structure, and a long history of updates.

Prehistory

The Simple Build Tool, as it used to be called, was originally released in 2008. When the first version 0.3.2 was published, code was hosted in the now almost defunct Google Code, and become the de facto tool for another project popular at the time: Lift. The first surviving commit after the migration to Github can be found here, belonging to version 0.5.0. SBT offered a way to build Scala applications with a configuration file written in pure Scala, supported subprojects, ScalaTest, Specs and ScalaCheck, and most importantly, managed dependencies via Apache Ivy and some support for accessing Maven Repos.
SBT 0.6 / 0.7.x relied on Scala 2.7, but could compile 2.8 and it was incredibly fragile it the relationship with the Scala Compiler. Any version change would require lots of fiddling around to get it to work (as documented here). SBT was achieving fame by being documented in the Twitter Scala School

Middle Ages

Migration from 0.7 to 0.10 in 2011 meant that SBT was itself built with Scala 2.8. A large re-organization meant lots of breaking changes, including the move the the .sbt build file configuration, a new Task engine, improved Plugin system, moving to Github, among others. This structure has been largely preserved since, despite many internal changes.
From version 0.11, just a few months after the release of 0.10, SBT itself moved to Scala 2.9, dropping support for Scala 2.7, freeing itself from the legacy scala collection API and lots of quirks. It also meant a large overhaul of the settings API, and removing the local Maven repo, and changing resolvers while moving infrastructure to different organization, once more causing pain for existing users.

Modern Times

The following year, for Scala Days 2012 saw the release of 0.12 (no pun intended?). Some of the benefits included incremental compilation, cross compilation for different scala versions (important to library writers having to support multiple binary versions of Scala), a launcher that was able to launch previous versions, improved documentation and other fixes. It also saw the introduction of the repositories file for global configuration and changes to the plugin structure. Things generally worked better than before, but migration was still a pain. The plugin ecosystem starts to flourish.

Industrial Age

SBT 0.13 was more a refinement that a revolution like previous iterations. SBT itself moved to Scala 2.10, and took advantage of the many new features available in the language. Tasks benefited from the improved macro support. The new syntax is implemented by making :=, +=, and ++= macros and making these the only required assignment methods. Documentation also vastly improved, thanks in part to better tooling and the progressive migration towards Typesafe. Up to version 0.13.1, Mark Harrah was the lead developer, but he decided to move on, and Josh Suereth became the main committer for SBT, who released 0.13.2 at the beginning of 2014.

Digital Age

From 0.13.5+ (currently .6, .7 and soon .8), every release in the Tech Previews for SBT 1.0 focused on improving existing functionality but also making SBT generally more usable and faster. New Typesafe employees Grzegorz Kossakowski (@gkossakowski) and Eugene Yokota made lots of contributions, such as: Auto Plugins: A simpler way to write plugins with dependencies between them. Better Incremental Compiler: Agressively cache and avoid recompiling, speeding up the build, particularly when macros are involved. Cached resolution: an improvement on the previous consolidated resolution cache resolutions for multi projects, which greatly helps speeds up update tasks since Ivy will not see dependencies across projects.

So what is SBT?

SBT is a number of different technologies under a common group. Lets dissect them.
SBT Launcher: Starts the SBT main process. Internally uses Scala 2.10 to update itself. All files in the project/ directories belong to it. (Launcher Architecture). This is quite possible the weakest link. The Launcher allows you to boot up the SBT Console. Relies on a multitude of shell scripts and configuration files, with quite some room for misconfiguration.
SBT Builder: The core of SBT, the reactive build tool. Parallel by default (tasks that don’t have any unresolved dependencies left will be run in parallel). SBT uses Apache Ivy for dependency resolution instead of Maven’s Aether. Recent versions of SBT also do dependency resolution caching(mainly to speed up Ivy queries), and aggressive compiled code caching. Since the Scala compiler has so much work to do, any speed up is greatly beneficial. Multi projects really help here, where the best practice is to actually create the smallest possible projects, and as many as possible. A great feature of SBT is that it works as a graph instead of a pipeline. It can run tests of already compiled projects while compiling others. This can easily max out the CPU because SBT will aggressively run tasks in parallel, using all available Cores.
SBT Incremental Compiler. SBT goes to great lengths to avoid recompiling. It tracks source dependencies at the granularity of source files. For each source file, sbt tracks files which depend on it directly. If the interface of classes, objects or traits in a file changes, all files dependent on that source must be recompiled. sbt does not instead track dependencies to source code at the granularity of individual output .class files, as one might hope. Doing so would be incorrect, because of some problems with sealed classes. See the documentation for more details.
Zinc: SBT standalone version of the incremental compiler, available for other build tools. Used by the Maven Scala plugin.

SBT Server: Went through several faces. Started as FSC (Fast Scala Compiler). It currently runs a Nailgun server which is keeps it running and warm between executions. See how this affected the design of the Scala plugin for Intellij Idea 12 onwards.

SBT Core Next: Set of reusable auto plugins. These new APIs were mostly created to enable sbt server, but they have null or fallback behavior in traditional non-server sbt, so they are safe to use unconditionally. The key one to note here is: BackgroundRunPlugin: This plugin introduces the concept of background jobs, which are threads or processes which exist in the background (outside of any task execution). Jobs can be managed similar to OS processes

sbt-remote-control: Converts sbt from a command line tool into a general backend for any kind of user interface, command line or GUI. See here for more (potentially out of date) information on the Client Server split

Activator: The Typesafe Activator is soon becoming the clean face of SBT. Activator is meant to become some kind of REPL/Play Console/Akka Typesafe Console/SBT/Nailgun/FSC/IDE/Kickstarter jack-of-all-trades tool. Large changes have been made, particularly to Play Framework, to extract out the Play Console and externalise those services.

Summary:

SBT is now a mature tool, getting ready for a 1.0 release. In its 6 years history it has shaped the user experience of most Scala users. Typesafe is keen to improve SBT, and brand the Activator as the new build UI for the Reactive Platform. SBT is nowadays a very useful build tool, so if you have had some bad experiences in the past, is time to look again.

Building Scala apps with Maven and ScalaTest

Of the many Scala projects I know (particularly on Github), most rely on SBT for the build. However, no matter how convenient SBT is, some projects are more easily handled by Maven, even if it is because a lot of the infrastructure require to deploy large projects, is already managed by it.

I personally prefer SBT, in fact, you can use this project for quick Scala + SBT + IDEs setup.  But for the projects I have at work, Maven is used, so I need to have a decent project file.

But as soon as you start trying to build something, you realise that the build systems in Scala still are in beta stages. The amount of options, poor documentation, and conflicting views certainly doesn't help:
SBT (with all its versions), two Maven plugins, Intellij, the ever changing Scala IDE plugin and at least two main testing frameworks, ScalaTest and Specs. Oh, and don't forget Java / Scala mixed projects. It can certainly alienate a newcomer.

In particular, the two maven plugins: maven-scala-plugin, a scala-maven-plugin, have plenty of different posts on how to configure them, most if not all of them at various stages of decay and rot, even though some of this posts are fairly recent. Or what happens a lot of the time, only fragments are posted, usually with old versions referenced.
The project to use should be m2eclipse-scala
Enough with the problems: Here's a simple, complete, and hopefully up to date example (At least at the time of writing):
Use my github project minimal-scala-maven, to make sure the latest version,  otherwise this is also going to decay soon



It's using ScalaTest 2.0.M4 although is not a requirement. ScalaTest is going to be the an umbrella framework, capable of plugging other testing frameworks on top if need be. A Scala IDE plugin is well advanced too, so quite important. Beware, scalatest is supposed to provide a new maven plugin, so stay tuned.

This project should work well with latest 11.1.3 version of Intellij or Scala IDE 2.1 M2 (even tough the maven plugin will complain about lifecycle execution, compiles fine). This last bit I've managed to remove in the past, but I gave up trying to fix every issue. It requires Maven 3.

See some posts for other working approaches with older plugins

Let me know if it works for you, and you have any suggestions on improvement, in particular around Scala IDE.

The JVM finally moving forward

After almost a year of delay, Java 7 has been officially released for the Mac. SeeJava 7 Mac OS X download and README.

In order to use it in Eclipse and others, bear in mind that the PATH is now:

/Library/Java/JavaVirtualMachines/1.7.0.jdk/Contents

This may be inaccesible to Eclipse, but you can paste the whole line, or create a symlink. It contains the sources, but no Javadocs.

This is a huge milestone, which marks the beginning of the end for Java 6, which was released 6 years ago. Technologies like Scala have soared since. But now with JDK 7 been readily available in all platforms, and with many bugfixes since the original release, it's time to upgrade. Last step will be JDK7u6 when the JRE will be fully availabe. See my previous post on Java 7, 8 and beyond

Update: After running a couple of scripts, I ran into this issue with UnknownHostException in JDK7

See this snippet:

This is my /etc/hosts

127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost
fe80::1%lo0 localhost
192.168.1.3 loki

Java 7, 8 and Beyond

Java 7 update 4 developer preview is available to download from OpenJDK website. Why is this any important? because is the first candidate release of the official JDK released by Oracle since Apple decided to stop shipping a custom build.

This also paves the way for further releases of Java 7 and ultimately Java 8 given the imminent end of life of Java 6.

Are you using Java 7 already? I can see it been similar to Java 5. Few people actually using it, and instead moving directly to Java 8, or feel pushed by it, to move into the last known stable version (in this case, 7).

Java 8 will bring lots of new features making the language a lot closer to what Scala delivers today, and is been heavily influenced by its success. See Simon Ritter's presentation and Fredrik Ohrstrom for more details. Better still... get yourself a build !

See Early Draft of Project Lambda by Brian Goetz. More info on Project Lambda for Java 8

More updates on Lambda, this time is on collections: Brian Goetz update on Streams for Java 8. I particularly like to stress this point on his blog post:

we will pursue an evolutionary strategy of adding extension methods to existing interfaces (such as Collection, List, or Iterable), or perhaps to new interfaces (such as 'Stream') that are retrofitted onto existing classes, enabling many of the desired idioms without making people trade in their trusty ArrayLists and HashMaps. (This is not to say that Java will never have a new Collections framework; clearly there are limitations with the existing Collections framework beyond simply not being designed for lambdas. Creating a new-and-improved collections framework is a fine candidate for consideration in a future version of the JDK.)

Even with Java8, Scala will continue to be ahead of Java, offering a more powerful and complete solution. The large change that was moving from Scala 2.7 to 2.8 in terms of upgrading collections, will have to wait at least until Java 9 to take effect. Because of this, Scala is a technology to stay, and lead the way to a functional revolution.

Good Days to be Scala developer

Next week is Scala Days London 2012, a 2 days marathon with some elite hackers. It's the first time I'm attending and really looking forward to it. The topics on discussion are extremely exciting. Distributed and cloud computing,  functional theory, Akka, Play Framework, Spark, Finagle, and obviously, cool production scenarios and war stories.

Typesafe Stack 2.0 brought so many cool things that takes a lot of time just to catchup, so really interested to attend talks around Akka 2.0 and Play 2.0.

The emphasis that Typesafe and the community are putting on tool support is paying dividends.

SBT has finally become a stable build environment and the community is thriving with an ever growing list of plugins. Intellij IDEA support is great and keeps expanding.

Today Scala 2.9.2 was released together with Scala IDE 2.1 M1, which pack lots of nice features like move refactoring, implicates highlighting, scala debugger and more!. And the roadmap to the final release looks great. This shows how Scala itself has matured, and how the whole ecosystem is taking life on its own.

That provides a lot of confidence not just to developers, but to small and large companies, that are increasingly considering Scala, as evidenced by the growing number of quality scala jobs available.

Exciting Scala Days !!

Understanding the JVM, concurrency and GC

A great collection of links and presentations on Java Virtual Machine and why understanding how it works is critical.

First, let's start with HotSpot, the default JVM. This is a mandatory intro into the responsibilities of the VM. Next, learning about the some of the most common optimizations that the JIT compiler has available, HotSpot FAQs, and method inlining

Now we are going to go down one abstraction layer, and jump into the hardware. A great presentation by Jevgeni Kabanov, by far the simplest to undestand: Do you really get Memory? I find this presentation a must see before jumping into the more hardcore versions that follow. In this talk, Jevgeni presents a simplified Java CPU model, showing the CPU, Memory, and more fundamentally, Caches.

Armed with this knowledge, we are then ready to take on the mammoth paper "What every programmer should know about memory by Ulrich Drepper. This is very low level at times, so worth watching Cliff Click's presentation: A Crash Course on Modern Hardware, which summarises it nicely.

Interesting article also on Memory Barriers and obviously, it wouldn't be complete without Doug Lea's concurrency in Java presentation. Always a good read, although really extensive, the Java Language Specification, Third Edition

Finally, I have reserved it for the last, Gil Tene's excellent Understanding Java Garbage Collection. Brings in lots of points that i didn't know about in my previous post The Art of GC Tuning. In particular, I really enjoyed the discussion around Application Memory Wall, and how our applications seem to range between 1-4 GBs, but rarely take advantage of increasing hardware power, and what are the main problems behind it. He shows how the Azul Zing VM can easily cope with 1/2 TB VMs without pausing.

That's quite a lot for now!

Rename a GIT project in GitHub

It is now supported to rename projects in GitHub.

Go to project admin, and rename.

Then, in your local .git project, simple edit your '.git/config' file

[remote "origin"]
        url = g...@github.com:fractal/old-name.git
        fetch = +refs/heads/*:refs/remotes/origin/*
...

Now just change old-name to new name, save and push