Wednesday 18 March 2015

A Brief History of SBT

Any developer that wrote more than a few lines of Scala has eventually heard of SBT. SBT is an efficient build tool with lofty functional principles, an uncommon structure, and a long history of updates.

Prehistory

The Simple Build Tool, as it used to be called, was originally released in 2008. When the first version 0.3.2 was published, code was hosted in the now almost defunct Google Code, and become the de facto tool for another project popular at the time: Lift. The first surviving commit after the migration to Github can be found here, belonging to version 0.5.0. SBT offered a way to build Scala applications with a configuration file written in pure Scala, supported subprojects, ScalaTest, Specs and ScalaCheck, and most importantly, managed dependencies via Apache Ivy and some support for accessing Maven Repos.
SBT 0.6 / 0.7.x relied on Scala 2.7, but could compile 2.8 and it was incredibly fragile it the relationship with the Scala Compiler. Any version change would require lots of fiddling around to get it to work (as documented here). SBT was achieving fame by being documented in the Twitter Scala School

Middle Ages

Migration from 0.7 to 0.10 in 2011 meant that SBT was itself built with Scala 2.8. A large re-organization meant lots of breaking changes, including the move the the .sbt build file configuration, a new Task engine, improved Plugin system, moving to Github, among others. This structure has been largely preserved since, despite many internal changes.
From version 0.11, just a few months after the release of 0.10, SBT itself moved to Scala 2.9, dropping support for Scala 2.7, freeing itself from the legacy scala collection API and lots of quirks. It also meant a large overhaul of the settings API, and removing the local Maven repo, and changing resolvers while moving infrastructure to different organization, once more causing pain for existing users.

Modern Times

The following year, for Scala Days 2012 saw the release of 0.12 (no pun intended?). Some of the benefits included incremental compilation, cross compilation for different scala versions (important to library writers having to support multiple binary versions of Scala), a launcher that was able to launch previous versions, improved documentation and other fixes. It also saw the introduction of the repositories file for global configuration and changes to the plugin structure. Things generally worked better than before, but migration was still a pain. The plugin ecosystem starts to flourish.

Industrial Age

SBT 0.13 was more a refinement that a revolution like previous iterations. SBT itself moved to Scala 2.10, and took advantage of the many new features available in the language. Tasks benefited from the improved macro support. The new syntax is implemented by making :=, +=, and ++= macros and making these the only required assignment methods. Documentation also vastly improved, thanks in part to better tooling and the progressive migration towards Typesafe. Up to version 0.13.1, Mark Harrah was the lead developer, but he decided to move on, and Josh Suereth became the main committer for SBT, who released 0.13.2 at the beginning of 2014.

Digital Age

From 0.13.5+ (currently .6, .7 and soon .8), every release in the Tech Previews for SBT 1.0 focused on improving existing functionality but also making SBT generally more usable and faster. New Typesafe employees Grzegorz Kossakowski (@gkossakowski) and Eugene Yokota made lots of contributions, such as: Auto Plugins: A simpler way to write plugins with dependencies between them. Better Incremental Compiler: Agressively cache and avoid recompiling, speeding up the build, particularly when macros are involved. Cached resolution: an improvement on the previous consolidated resolution cache resolutions for multi projects, which greatly helps speeds up update tasks since Ivy will not see dependencies across projects.

So what is SBT?

SBT is a number of different technologies under a common group. Lets dissect them.
SBT Launcher: Starts the SBT main process. Internally uses Scala 2.10 to update itself. All files in the project/ directories belong to it. (Launcher Architecture). This is quite possible the weakest link. The Launcher allows you to boot up the SBT Console. Relies on a multitude of shell scripts and configuration files, with quite some room for misconfiguration.
SBT Builder: The core of SBT, the reactive build tool. Parallel by default (tasks that don’t have any unresolved dependencies left will be run in parallel). SBT uses Apache Ivy for dependency resolution instead of Maven’s Aether. Recent versions of SBT also do dependency resolution caching(mainly to speed up Ivy queries), and aggressive compiled code caching. Since the Scala compiler has so much work to do, any speed up is greatly beneficial. Multi projects really help here, where the best practice is to actually create the smallest possible projects, and as many as possible. A great feature of SBT is that it works as a graph instead of a pipeline. It can run tests of already compiled projects while compiling others. This can easily max out the CPU because SBT will aggressively run tasks in parallel, using all available Cores.
SBT Incremental Compiler. SBT goes to great lengths to avoid recompiling. It tracks source dependencies at the granularity of source files. For each source file, sbt tracks files which depend on it directly. If the interface of classes, objects or traits in a file changes, all files dependent on that source must be recompiled. sbt does not instead track dependencies to source code at the granularity of individual output .class files, as one might hope. Doing so would be incorrect, because of some problems with sealed classes. See the documentation for more details.
Zinc: SBT standalone version of the incremental compiler, available for other build tools. Used by the Maven Scala plugin.

SBT Server: Went through several faces. Started as FSC (Fast Scala Compiler). It currently runs a Nailgun server which is keeps it running and warm between executions. See how this affected the design of the Scala plugin for Intellij Idea 12 onwards.

SBT Core Next: Set of reusable auto plugins. These new APIs were mostly created to enable sbt server, but they have null or fallback behavior in traditional non-server sbt, so they are safe to use unconditionally. The key one to note here is: BackgroundRunPlugin: This plugin introduces the concept of background jobs, which are threads or processes which exist in the background (outside of any task execution). Jobs can be managed similar to OS processes

sbt-remote-control: Converts sbt from a command line tool into a general backend for any kind of user interface, command line or GUI. See here for more (potentially out of date) information on the Client Server split

Activator: The Typesafe Activator is soon becoming the clean face of SBT. Activator is meant to become some kind of REPL/Play Console/Akka Typesafe Console/SBT/Nailgun/FSC/IDE/Kickstarter jack-of-all-trades tool. Large changes have been made, particularly to Play Framework, to extract out the Play Console and externalise those services.


Summary:

SBT is now a mature tool, getting ready for a 1.0 release. In its 6 years history it has shaped the user experience of most Scala users. Typesafe is keen to improve SBT, and brand the Activator as the new build UI for the Reactive Platform. SBT is nowadays a very useful build tool, so if you have had some bad experiences in the past, is time to look again.