A Brief History of SBT

Any developer that wrote more than a few lines of Scala has eventually heard of SBT. SBT is an efficient build tool with lofty functional principles, an uncommon structure, and a long history of updates.

Prehistory

The Simple Build Tool, as it used to be called, was originally released in 2008. When the first version 0.3.2 was published, code was hosted in the now almost defunct Google Code, and become the de facto tool for another project popular at the time: Lift. The first surviving commit after the migration to Github can be found here, belonging to version 0.5.0. SBT offered a way to build Scala applications with a configuration file written in pure Scala, supported subprojects, ScalaTest, Specs and ScalaCheck, and most importantly, managed dependencies via Apache Ivy and some support for accessing Maven Repos.
SBT 0.6 / 0.7.x relied on Scala 2.7, but could compile 2.8 and it was incredibly fragile it the relationship with the Scala Compiler. Any version change would require lots of fiddling around to get it to work (as documented here). SBT was achieving fame by being documented in the Twitter Scala School

Middle Ages

Migration from 0.7 to 0.10 in 2011 meant that SBT was itself built with Scala 2.8. A large re-organization meant lots of breaking changes, including the move the the .sbt build file configuration, a new Task engine, improved Plugin system, moving to Github, among others. This structure has been largely preserved since, despite many internal changes.
From version 0.11, just a few months after the release of 0.10, SBT itself moved to Scala 2.9, dropping support for Scala 2.7, freeing itself from the legacy scala collection API and lots of quirks. It also meant a large overhaul of the settings API, and removing the local Maven repo, and changing resolvers while moving infrastructure to different organization, once more causing pain for existing users.

Modern Times

The following year, for Scala Days 2012 saw the release of 0.12 (no pun intended?). Some of the benefits included incremental compilation, cross compilation for different scala versions (important to library writers having to support multiple binary versions of Scala), a launcher that was able to launch previous versions, improved documentation and other fixes. It also saw the introduction of the repositories file for global configuration and changes to the plugin structure. Things generally worked better than before, but migration was still a pain. The plugin ecosystem starts to flourish.

Industrial Age

SBT 0.13 was more a refinement that a revolution like previous iterations. SBT itself moved to Scala 2.10, and took advantage of the many new features available in the language. Tasks benefited from the improved macro support. The new syntax is implemented by making :=, +=, and ++= macros and making these the only required assignment methods. Documentation also vastly improved, thanks in part to better tooling and the progressive migration towards Typesafe. Up to version 0.13.1, Mark Harrah was the lead developer, but he decided to move on, and Josh Suereth became the main committer for SBT, who released 0.13.2 at the beginning of 2014.

Digital Age

From 0.13.5+ (currently .6, .7 and soon .8), every release in the Tech Previews for SBT 1.0 focused on improving existing functionality but also making SBT generally more usable and faster. New Typesafe employees Grzegorz Kossakowski (@gkossakowski) and Eugene Yokota made lots of contributions, such as: Auto Plugins: A simpler way to write plugins with dependencies between them. Better Incremental Compiler: Agressively cache and avoid recompiling, speeding up the build, particularly when macros are involved. Cached resolution: an improvement on the previous consolidated resolution cache resolutions for multi projects, which greatly helps speeds up update tasks since Ivy will not see dependencies across projects.

So what is SBT?

SBT is a number of different technologies under a common group. Lets dissect them.
SBT Launcher: Starts the SBT main process. Internally uses Scala 2.10 to update itself. All files in the project/ directories belong to it. (Launcher Architecture). This is quite possible the weakest link. The Launcher allows you to boot up the SBT Console. Relies on a multitude of shell scripts and configuration files, with quite some room for misconfiguration.
SBT Builder: The core of SBT, the reactive build tool. Parallel by default (tasks that don’t have any unresolved dependencies left will be run in parallel). SBT uses Apache Ivy for dependency resolution instead of Maven’s Aether. Recent versions of SBT also do dependency resolution caching(mainly to speed up Ivy queries), and aggressive compiled code caching. Since the Scala compiler has so much work to do, any speed up is greatly beneficial. Multi projects really help here, where the best practice is to actually create the smallest possible projects, and as many as possible. A great feature of SBT is that it works as a graph instead of a pipeline. It can run tests of already compiled projects while compiling others. This can easily max out the CPU because SBT will aggressively run tasks in parallel, using all available Cores.
SBT Incremental Compiler. SBT goes to great lengths to avoid recompiling. It tracks source dependencies at the granularity of source files. For each source file, sbt tracks files which depend on it directly. If the interface of classes, objects or traits in a file changes, all files dependent on that source must be recompiled. sbt does not instead track dependencies to source code at the granularity of individual output .class files, as one might hope. Doing so would be incorrect, because of some problems with sealed classes. See the documentation for more details.
Zinc: SBT standalone version of the incremental compiler, available for other build tools. Used by the Maven Scala plugin.

SBT Server: Went through several faces. Started as FSC (Fast Scala Compiler). It currently runs a Nailgun server which is keeps it running and warm between executions. See how this affected the design of the Scala plugin for Intellij Idea 12 onwards.

SBT Core Next: Set of reusable auto plugins. These new APIs were mostly created to enable sbt server, but they have null or fallback behavior in traditional non-server sbt, so they are safe to use unconditionally. The key one to note here is: BackgroundRunPlugin: This plugin introduces the concept of background jobs, which are threads or processes which exist in the background (outside of any task execution). Jobs can be managed similar to OS processes

sbt-remote-control: Converts sbt from a command line tool into a general backend for any kind of user interface, command line or GUI. See here for more (potentially out of date) information on the Client Server split

Activator: The Typesafe Activator is soon becoming the clean face of SBT. Activator is meant to become some kind of REPL/Play Console/Akka Typesafe Console/SBT/Nailgun/FSC/IDE/Kickstarter jack-of-all-trades tool. Large changes have been made, particularly to Play Framework, to extract out the Play Console and externalise those services.

Summary:

SBT is now a mature tool, getting ready for a 1.0 release. In its 6 years history it has shaped the user experience of most Scala users. Typesafe is keen to improve SBT, and brand the Activator as the new build UI for the Reactive Platform. SBT is nowadays a very useful build tool, so if you have had some bad experiences in the past, is time to look again.

Good Days to be Scala developer

Next week is Scala Days London 2012, a 2 days marathon with some elite hackers. It's the first time I'm attending and really looking forward to it. The topics on discussion are extremely exciting. Distributed and cloud computing,  functional theory, Akka, Play Framework, Spark, Finagle, and obviously, cool production scenarios and war stories.

Typesafe Stack 2.0 brought so many cool things that takes a lot of time just to catchup, so really interested to attend talks around Akka 2.0 and Play 2.0.

The emphasis that Typesafe and the community are putting on tool support is paying dividends.

SBT has finally become a stable build environment and the community is thriving with an ever growing list of plugins. Intellij IDEA support is great and keeps expanding.

Today Scala 2.9.2 was released together with Scala IDE 2.1 M1, which pack lots of nice features like move refactoring, implicates highlighting, scala debugger and more!. And the roadmap to the final release looks great. This shows how Scala itself has matured, and how the whole ecosystem is taking life on its own.

That provides a lot of confidence not just to developers, but to small and large companies, that are increasingly considering Scala, as evidenced by the growing number of quality scala jobs available.

Exciting Scala Days !!

First Scala Step - Simple SBT template

I've created a simple project to clone when you need to get up and running really quickly using the latest Scala tools.

It provides:

•  Scala 2.9.1
•  SBT 0.11.2
•  Specs2 1.6
•  Mockito 1.8.5
•  Eclipse
•  IntelliJ IDEA

Clone first-scala-step and start using it! 

Alternatively, just today, coolscala has also pushed a similar project to kickstart using Akka 1.3-RC1+ ScalaTests : start-akka

Exploring Scala Collections

This weekend I've been trying to write some microbenchmarks and discovered a large differences between different implements of the same problem: create a list of 10 million objects and print only the ones that are modulo of a million.

This is the code in question. It runs in 141 milliseconds requiring no more than 27 mbs JVM (the smallest i could create).

As shown in this StackOverFlow question, a range followed by a filter operation can even beat the performance of an array of primitives in Java. The space required is smaller, and the amount of time to execute it is barely comparable, whereas any other approach was in between one and two orders of magnitude worse, depending the case.

I learnt several things with this simple exercise, from Ranges, Array (compile to primitives, no need for Wrappers), and performance monitoring.

The costs of concurrency and the actor model

I was interested to understand more about how actors are implemented in Erlang. In an old but still relevant article, an argument is raised:

These days every third grader knows that processes are expensive. Ok, so we're not using real processes, we're using threads, but if you constantly create, destroy, and switch between thousands of threads you'll bring any system to a crawl. Threads are expensive for a number of reasons. They take a long time and a lot of memory to create because the operating system needs to set up many things (the stack, internal data structures, etc.) for them to work. They take a long time to destroy because all the resources they consumed need to be freed. And they take a long time to switch between because unloading registers, storing them, loading them back, and flipping the stacks is complicated business.

So how expensive it really is to create a Thread in java these days?

Peter Lawrey, writer of the excellent Vanilla Java blog, tell us that roughly, creating a Thread takes
about 70μ (microseconds), which is fairly expensive, but not prohibitive. The size of the stack can be configured by command line argument to the JVM, with a default of 512kb. According to some Mac OS X 10.5 docs, a thread takes about 90μ, which is kind of validates his results.

As it was pointed out in the StackOverflow discussion

Creating threads is expensive if you're planning on firing 2000 threads per second, every second of your runtime. The JVM is not designed to handle that. If you'll have a couple of stable workers that won't be fired and killed over and over, relax.

This is slightly different in Erlang, where everything is a Process, so the model leads you to that situation.

Essentially, Erlang uses a kind of lightweight processes, akin to green threads (VM scheduled pseudo-threads as opposed to real threads). Since they only consist of a mailbox implemented as a queue, it takes about 300 words to build one, meaning we can create thousands, even millions of them, so the only constrain is memory.

I read a great post on Erlang lightweight-process actor concurrency and it's inherent problems in terms of message passing costs, i.e., having to serialize the entire message between processes, even within the same node. The lack of zero-copy messaging between actors, even within the same node, can really hurt performance. It also talks a great deal on how the BEAM VM and HiPE JIT compiler aren't anywhere near as good as the industry leader JVM (or his favoured Azul Systems "pauseless" VM).

On that basis, after years of hardcore work on Erlang, he decants for Erjang (using Java's Killim actors) or Clojure, possibly
due to been more functional and lisp syntax. I wonder why Scala wasn't mentioned? Akka
offers the same principles of Erlang actors, but implemented in a much more robust way and running on top of the JVM.

Scala actors, particularly Akka actors, use configurableexecutors that efficiently reuse threads (and in Akka 2.0, this will be declaratively configured). Thus, i can see Akka
as supersiding Erlang, an the heir of the throne in actors concurrency.

Scala Options: Type and Null Safety

Options are great because they allow you to circumvent the $1 billion bug (Hoare).

Several posts discuss the great values of Options. It was Graham Tackley who really made me click when he presented Options as collections with zero or one elements in it.

Yesterday I was trying to make good use of this idea, and came with the problem of how retain only the elements that actually contain a value. Since you cannot call get on a None for obvious reasons, there must be an easy way to retrieve only the values.

Here's what i found:

I've transformed a List[Option[String]] with possible Nones in it, to a List[String].

How about this then?

Which is basically doing the operation that I've just done before, call flatMap(_.toList)

This makes passing around options a really valuable thing, because you have safety in your operations with minimal effort.

Very useful Scala Option cheatlist by Tony Morris

Building Scala applications with SBT + Idea

Intellij IDEA is a great tool for Scala Developers. A couple of days ago I attended a great event of the London Scala User Group where, Richard Hallaway, Phill Wills and Maciej did a very good intro to this tool. This guide explains how to quickly leverage a couple of tools to setup a new Scala project in minutes.

First of all, start downloading right now the latest build of Idea X community edition for your platform, this will probably be ready by the time we finish with the setup. You'll also need the Scala and SBT plugin.

There are basically two ways to setup your idea projects with SBT. The first would be using Heiko Seerbergers guide to setup the project from Idea. I prefer to do it the way that Mikko Peltonen suggests... SBT-Idea.

First, you'll need the great SBT (Simple Build Tool), which is a build tool written in Scala and for Scala. Its power lies in that it relies on convention over configuration and giving you full control over your builds. No longer long XML configurations + Shell scripts. SBT can do everything from one single script. Furthermore, builds can pickup only the changes you made, and recompile and run only those tests. Check it out!

After you setup SBT  , create a new project:

sbt

Name:

Organization:

Scala: 2.8.1

sbt: 0.7.4

this will create most of your project.

then exit the console, go to project folder, and create a folder build and inside, a scala file such as this:

Go back to the sbt console, type update, and it will pull the dependencies.

Use the SBT processor as specified here:

> *sbtIdeaRepo at http://mpeltonen.github.com/maven/
 > *idea is com.github.mpeltonen sbt-idea-processor 0.1-SNAPSHOT 
 ...
 > update
 ...
 > idea
 ...

this will create your project. Now, import in IDEA. All your SBT structure will be recognized.

Now let's setup the FSC (Scala Fast Compiler Server). SBT is great for quick reloads, but i still prefer to have fast compilation within the IDE.

For that, just go to Edit Configurations in your run targets in the toolbar, and create a new Scala Compilation Server using the following details. I had issues in the past when ticking the box shared configurations but it probably has been fixed already.

That's all you need ! Super fast compilation within Idea. And remember if you run your commands in sbt with a tilde ~ they will automatically run every time you switch back to the sbt console and it detects a file change.

Finally, don't forget your .gitignore

.idea/
lib_managed/
project/boot/
project/build/target/
target/

Scala 2.8.1 released

Plenty of bug fixes (more than 100 in 3 months!). Great to see a very active community, kicking the tires and gathering to solve problems. Furthermore, It's fully binary compatible with 2.8, which means no need to match library versions as it happened with the RCs for 2.8. Also, the improvements to Scaladocs make it look very snappy

All in all, we can see the the Scala language growing at a fast pace. Many new books are coming, including Phillip Haller's Actors in Scala and three others by Manning "in Action" series. Browse the complete list of books if you are interested.

Scala just pushed Play in web development

Until recently, Scala only had Lift has a successful web framework. Meanwhile, the ruby gang has hundreds of them, more importantly ruby on rails, Sinatra and others. But things are changing fast in the scala world. While scalatra deals with Sinatra, there was nothing quite like ror. Enter Play Framework for Scala. Module history here

I must admit that this 11 pages guide to build from scratch an entire blog system, together with captcha, user administration console, open id authentication, using a clean RESTful interface, together with autogenerated unit, integration and selenium tests.... well, this is just absolutely fantastic. That's what i call a quality framework documentation! Scala just pushed Play > !

If you add free JRebel free licenses to Scala developers by ZeroTurnaround, Scala is automatically an ultra productive environment for web development!

Great effort! Can't wait for Play Framework 1.1!

Scala community growing ever faster

I'm becoming more involved with the Scala community and it looks very, very promising. The Scala Lift-Off event in London was a success, as demonstrated by richard's blog and maciek's notes on day 1  and day 2

Scala 2.8 was released in July, and already has an impressive list of bugfixes in 2.8.1. This demostrates the commitment from the community. Furthermore, Martin Odersky announced Scala Solutions, a new venture to offer commercial support and training.

The two biggest proposals of the event were Lift, which released the version 2.1 of their framework. Tim, who is writing a book on Lift made some really interesting comments. Particularly, I learnt about Squeryl, a fantastic framework to construct SQL queries in a friendly Scala DSL syntax, offering full compiler support ... and more importantly, refactoring.

Akka, the concurrency framework is close to the much awaited 1.0 release. Jonas Boner has made many presentations since then, including one at LinkedIn , which, by the way, uses Scala intensively with both Norbert and Scalatra, a Sinatra-like web framework in Scala

Graham Tackley, and ex-colleague of mine, did a nice presentation on The Guardian move to (and love of) Scala

Videos are available here

For great presentations, including some upcoming features in 2.9, have a look at the  Scala Days 2010 website. Parallel collections, scala refactorings for Eclipse, and much more.

Exciting days for the Scala community !

In the meantime, the demand for Scala in my current job is growing exponentially, so I'm currently preparing some presentations on the subject. Stay tuned!

Scala 2.8.0 FINAL released

The long awaited Scala 2.8.0 has been released today (just in time for my dissertation!)

The list of new features is gigantic, and thoroughly deserves a 3.0 release, but due to distribution issues, the version upgrade is just minor.

Most distinguished changes, the complete revamp of the collections API, named and default arguments, package objects, enhanced event based actors better documentation, thousands of bug fixes, Scala Eclipse IDE... too many to enumerate! Detailed changeset in the Scala website.

But most significantly, it means that all the scala libraries can now be compiled against a stable version, due to binary incompatibility even across minor releases.

This is a great day for the growing Scala community!

Comparing functional programming with JDK7 Lambdas and Scala

It seems that since Oracle bought Sun, they want to start breathing new life into Java. Out of the blue, Project Lambda seems to be pushing the spec for what would end up been functional idioms for Java.

The debate that the new syntax brings is captured in this InfoQ article on project Lambda.

compare Java 7's:


with Scala's




for my part, i prefer Scala's functional syntax, since I have a first class function that i can invoke. Brian Goetz (Concurrency guru) suggests that at the moment they are focusing in functionality rather than syntax, and that it will evolve.

Now, they better work a lot in order to improve this spaghetti:



For a move in depth review of the changes proposed, read this analysis on Project Lambda in InfoQ

The interesting point to make out of this is that Functional Programming is becoming more and more important and Java doesn't want to miss out. A few months ago, JDK7 was leaving closures out of the spec in order to ship a highly demanded final version. But now Oracle is in command, and they want to include FP in Java. This means that effectively Functional Programming is now becoming mainstream, supported in almost every major platform: F# in .Net, Scala, python, javascript, groovy, and now Java.

Simple build configuration for Scala 2.8 with Maven

There are plenty of blogs explaining how to run Scala with Maven, but, unfortunately, many of them are out of them, no less the scala-lang website, which I will try to update with this simple recipe.

The good thing about this configuration, is that is tested with both Eclipse and IntelliJ Idea 9.0.2 (open source)




The directory structure remains the same as for java, with the obvious difference:

src/main/scala
src/test/scala

Finally, add your package and test it with a Canary Test




JUnit 4 + Hamcrest matchers seems like a very simple combination that will get the job done without dealing with the binary incompatibility problems of Scala before the final release of 2.8. It seems that jars need to be compiled against the exact version of the compiler or they won't work. I don't yet know enough about this problem, but was significant enough to drive me away of using the most recent versions of ScalaTest and Specs for now. Thus, I'm sticking to
2.8.0.RC1 which seems to work in all IDEs.

At the moment i'm using Intellij Idea 9.0.2, which comes with a pretty good Scala plugin. Eclipse is usually my IDE of choice, but the plugin is in an early stage yet, since doesn't support any refactorings.

Another rather annoying problem was with Hamcrest, which doesn't have the 1.2 version available in Maven Repositories, but it comes with some nice matchers some I recommend you to download it from their website. It seems that the High priority bug to deploy it in Maven Repo Central has been open for... 3 years! I added the 35th request to get it closed :) Otherwise, just switch to 1.1 version, which is available from Maven Central.


Enjoy!

The end of the line

Ever wondered how to move to the start or the end of the line in the terminal in your MacBook? Ctrl+A and Ctrl+E respectively.

It also works on the Scala REPL, where you can quickly evaluate code snippets. Some useful tips here

Even while you surely know, home, end, page up and down are all accessible by combining the Fn + arrow keys.

Another end of the line shortcut is Shift+G in less. To tail that buffer do Shift+F (Ctrl+C to stop).

Enjoy!

Hello Scala World

Simple and concise. The thing to notice here is the keyword object. Scala has the built in concept of a Singleton, which simplifies design. All the static members are to be placed in the object companion, which replaces enums and acts as a default factory for your object. Brilliant!


To do enums, we have several alternatives:



or



Which can be used extensively with case classes.

Thanks to Carte Cole's syntax highlighting tutorial using SyntaxHighlighter

Vim syntax highlighting for Scala

Scala offers the flexibility to execute from command line (scala -e), through its REPL, as a script, or accessing the conventional main class. While REPL can be useful for quick hacks, the script has the advantage that it can be used for good shell scripts. On my Mac OS X Snow Leopard, (although it works on any unix) I configured vim as such:

curl http://lampsvn.epfl.ch/svn-repos/scala/scala-tool-support/trunk/src/vim/syntax/scala.vim -o .vim/syntax/scala.vim

curl http://lampsvn.epfl.ch/svn-repos/scala/scala-tool-support/trunk/src/vim/indent/scala.vim -o .vim/indent/scala.vim

curl http://lampsvn.epfl.ch/svn-repos/scala/scala-tool-support/trunk/src/vim/ftdetect/scala.vim -o .vim/ftdetect/scala.vim

then inside of vim, activate it:

:syntax on

Finally, to enable highlighting by default, add/modify your .vimrc and add the line

:syntax enable

Restart your console and you can now enjoy scala highlighting (although the color theme probably needs tweaking...)

Polyglot programmer

For years, Java has been considered 'the enterprise programming language', only contested by .Net. But since ruby on rails appeared, the development community started asking questions whether Java was the right choice in every case. A movement sparked by Andy Hunt and David Thomas following their book The Pragmatic Programmer led to the concept of Polyglot Programmer. They suggest that developers should 'speak' more than one language, and learn one every year, to stay fit, and adopt different paradigms such as functional programming (Lisp descendants such as Haskell) or the parallelism friendly Erlang.

Each language have pros and cons, but unlike Ruby, a handful of this new languages chose to profit from the large efforts invested on the JVM. It started off with
Groovy, a dynamic language that appealed Java developers due to its compatibility, Jython (Python interpreter), but moved on to JRuby, Scala and finally Clojure.


Scala has the added benefit that is binary compatible with Java, which means that is fully interoperable, and as such able to access its gigantic library.

From this year onwards, my goal is to become proficient in Scala, which I will be using for my dissertation. Stay tuned for more on Scala!