Scalafix - a Scala rewrite tool


0.4.2

Join the chat at https://gitter.im/scalacenter/scalafix

Scalafix is a Scala code rewriting tool and library. This effort follows the Scala Center Advisory Board proposal: Clarification of Scala to Dotty migration path. The long-term goal of the project is to automate migration from Scala 2.x to Dotty, a next-generation Scala compiler.

The name "scalafix" is inspired by gofix, a similar tool used to migrate code written in the Go programming language.

Installation


Scalafix can used either as a tool or a library. Integrations are currently limited to the command-line, but it may be possible to add IDE integrations in the future.

sbt-scalafix

The sbt-plugin is the recommended integration for semantic rewrites.
addSbtPlugin("ch.epfl.scala" % "sbt-scalafix" % "0.4.2")

scalafix-core

Scalafix can be used as a library to run custom rewrites.
libraryDependencies += "ch.epfl.scala" % "scalafix-core" % "0.4.2" cross CrossVersion.full

By using scalafix as a library, you have full control of how/when/where rewrites are written.

When writing custom rewrites, you should decide what API you need. There currently available APIs are:

Scalafix as a library is still under heavy development. Feedback is very welcome. Be prepared for breaking changes.

scalafix-cli

The recommended way to install the scalafix command-line interface is with coursier.
// coursier
coursier bootstrap ch.epfl.scala:scalafix-cli_2.11.11:0.4.2 -f --main scalafix.cli.Cli -o scalafix
./scalafix --help

// homebrew
brew install --HEAD olafurpg/scalafmt/scalafix
scalafix --help

// wget
wget -O scalafix https://github.com/scalacenter/scalafix/blob/master/scalafix?raw=true
./scalafix --help

Once the scalafix cli is installed, consult the --help page for further usage instructions

--help

scalafix 0.4.2+38-0877eb85
Usage: scalafix [options] [<file>...]
  --usage  
        Print usage and exit
  --help | -h  
        Print help message and exit
  --version | -v  
        Print version number and exit
  --verbose  
        If set, print out debugging inforation to stderr.
  --config | -c  <.scalafix.conf>
        File path to a .scalafix.conf configuration file.
  --config-str | -c  <imports.organize=false>
        String representing scalafix configuration
  --sourceroot  </foo/myproject>
        Absolute path passed to scalahost with -P:scalahost:sourceroot:<path>. Relative filenames persisted in the Semantic DB are absolutized by the sourceroot. Defaults to current working directory if not provided.
  --classpath  <entry1.jar:entry2.jar:target/scala-2.12/classes>
        java.io.File.pathSeparator separated list of directories or jars containing '.semanticdb' files. The 'semanticdb' files are emitted by the scalahost-nsc compiler plugin and are necessary for semantic rewrites like ExplicitReturnTypes to function.
  --rewrites | -r  <ProcedureSyntax OR file:LocalFile.scala OR scala:full.Name OR https://gist.com/.../Rewrite.scala>
        Rewrite rules to run.
  --stdout  
        If set, print fix to stdout instead of writing to file.
  --out-from  </shared/>
        Regex that is passed as first argument to fileToFix.replaceAll(outFrom, outTo)
  --out-to  </custom/>
        Replacement string that is passed as second argument to fileToFix.replaceAll(outFrom, outTo)
  --exclude  <core Foobar.scala>
        Space separated list of regexes to exclude which files to fix. If a file match one of the exclude regexes, then it will not get fixed. Defaults to excluding no files.
  --single-thread  
        If true, run on single thread. If false (default), use all available cores
  --no-sys-exit  
        If true, does not sys.exit at the end. Useful for example in sbt-scalafix
  --quiet-parse-errors  
        Don't report parse errors for non-explictly passed filepaths.
  --bash  
        Print out bash completion file for scalafix. To install on
          scalafix --bash > /usr/local/etc/bash_completion.d/scalafix # Mac
          scalafix --bash > /etc/bash_completion.d/scalafix           # Linux
  --zsh  
        Print out zsh completion file for scalafix. To install:
          scalafix --zsh > /usr/local/share/zsh/site-functions/_scalafix
Available rewrites: NoValInForComprehension, RemoveXmlLiterals, VolatileLazyVal, ProcedureSyntax, ExplicitUnit, DottyVarArgPattern, ExplicitReturnTypes, RemoveUnusedImports, Xor2Either, NoAutoTupling, NoExtendsApp

NOTE. The command line tool is mostly intended to be invoked programmatically
from build-tool integrations such as sbt-scalafix. The necessary fixture to run
semantic rewrites is tricky to setup manually.

Scalafix chooses which files to fix according to the following rules:
- when --syntactic is passed, then Scalafix looks for .scala files in the provided files/directories.
- by default, looks for .semanticdb files and matches them back to the original files.
  - if --classpath and --sourceroot are provided, then those are used to find .semanticdb files.
  - otherwise, Scalafix will automatically look for META-INF/semanticdb directories from the
    current working directory.

Examples (semantic):
  $ scalafix # automatically finds .semanticdb files and runs rewrite configured in .scalafix.conf.
  $ scalafix <directory> # same as above except only run on files in <directory>
  $ scalafix --rewrites RemoveUnusedImports # same as above but run RemoveUnusedImports.
  $ scalafix --classpath <foo.jar:target/classes> # explicitly pass classpath, --sourceroot is cwd.
  $ scalafix --classpath <foo.jar:target/classes> --sourceroot <directory>

Examples (syntactic):
  $ scalafix --syntactic --rewrites=ProcedureSyntax Code.scala # write fixed file in-place
  $ scalafix --syntactic --rewrites=ProcedureSyntax --stdout Code.scala # print fixed file to stdout
  $ cat .scalafix.conf
  rewrites = [ProcedureSyntax]
  $ scalafix Code.scala # Same as --rewrites ProcedureSyntax

Exit status codes:
 Ok=0
 UnexpectedError=1
 ParseError=2
 ScalafixError=4
 InvalidCommandLineOption=8
 MissingSemanticApi=16
 StaleSemanticDB=32

Configuration


Scalafix reads configuration from a file using HOCON syntax. I recommend you put a file .scalafix.conf into the root directory of your project.

rewrites

You can either run a pre-made that comes with scalafix or a custom rewrite that you write yourself.

Pre-made rewrites

Available rewrites are listed in Rewrites.
rewrites = [ProcedureSyntax] # No rewrites are run if empty.

Custom rewrites

Scalafix supports loading rewrites from a few different URI protocols.

scala:

If a rewrites is on the classpath, you can classload it with the scala: protocol.
rewrite = "scala:scalafix.rewrite.ProcedureSyntax"

file:

If a rewrites is written in a single file on disk, you can load it with the file: protocol.
rewrite = "file:readme/MyRewrite.scala" // from local file

http:

If a rewrite is written in a single source file on the internet, you can load it with the https: or http: protocol
rewrite = "https://gist.githubusercontent.com/olafurpg/fc6f43a695ac996bd02000f45ed02e63/raw/80218434edb85120a9c6fd6533a4418118de8ba7/ExampleRewrite.scala"

github:

If a rewrite is written in a single file and you want a short syntax, you can use the github: protocol for sharing your rewrite
// expands into Ok(https://raw.githubusercontent.com/org/repo/master/scalafix/rewrites/src/main/scala/fix/Repo_1_0_0.scala)
rewrite = "github:org/repo/1.0.0"

patches

For simple use-cases, it's possible to write custom rewrites directly .scalafix.conf.
patches.removeGlobalImports = [
  "scala.collection.mutable" // scala.meta.Importee
]
patches.addGlobalImports = [
  "scala.collection.immutable"
]
patches.replacements = [
  {
    from = "_root_.scala.Seq." // scala.meta.Symbol
    to = "Seq"
    additionalImports = [ "scala.collection.immutable.Seq" ]
  }
]
// Helper to see which symbols appear in your source files
debug.printSymbols = true
For more advanced use-cases, I recommend you use scalafix-core as a library.

All options

The following is the list of all available options along with their default values.
explicitReturnTypes.memberKind = []
explicitReturnTypes.memberVisibility = []
explicitReturnTypes.skipSimpleDefinitions = true
x = "{}"
patches.removeGlobalImports = []
patches.addGlobalImports = []
patches.replacements = []
reporter.outStream = java.io.PrintStream@599d85b3
reporter.minSeverity = "[34minfo[0m"
reporter.filter.".*"
reporter.includeLoggerName = false
parser = scala.meta.internal.parsers.ScalametaParser$$anon$202@2a6d89d1
debug.printSymbols = false
fatalWarnings = true
dialect = Scala211

Rewrites


Scalafix comes with a few rewrites out-of-the-box. These rewrites have been chosen to meet the long-term goal of scalafix to clarify the Scala to Dotty migration path. To create custom rewrites, see scalafix-core.

ExplicitReturnTypes

Dotty requires implicit vals and defs to explicitly annotate return types. The ExplicitReturnTypes rewrite inserts the inferred type from the compiler for implicit definitions that are missing an explicit return type. For example,
// before
implicit val tt = liftedType
// after
implicit val tt: TypedType[R] = liftedType

RemoveUnusedImports

This rewrite acts upon "Unused import" warnings emitted by the Scala compiler. See slick/slick/pulls#1736 for an example diff from running sbt "scalafix RemoveUnusedImports".

To use this rewrite:

// before
import scala.List
import scala.collection.{immutable, mutable}
object Foo { immutable.Seq.empty[Int] }

// after
import scala.collection.immutable
object Foo { immutable.Seq.empty[Int] }

Note. . This rewrite does a best-effort at preserving original formatting. In some cases, the rewritten code may be formatted weirdly

// before
import scala.concurrent.{
  CancellationException,
  TimeoutException
}
// after
import scala.concurrent.

  TimeoutException
It's recommended to use a code formatter after running this rewrite.

RemoveXmlLiterals

This rewrites replaces XML literals with a xml"" interpolator from scala-xml-quote project.
// tries to use single quote when possible
<div>{bar}</div>
xml"<div>${bar}</div>"

// multi-line literals get triple quote
<div>
  <span>{"Hello"}</span>
</div>
xml"""<div>
  <span>${"Hello"}</span>
</div>"""

// skips XML literals in pattern position
x match { case <a></a> => }
x match { case <a></a> => }

// replaces escaped {{ with single curly brace {
<div>{{</div>
xml"<div>{</div>"

ProcedureSyntax

"Procedure syntax" is not supported in Dotty. Methods that use procedure syntax should use regular method syntax instead. For example,
// before: procedure syntax
def main(args: Seq[String]) {
  println("Hello world!")
}
// after: regular syntax
def main(args: Seq[String]): Unit = {
  println("Hello world!")
}

VolatileLazyVal

Adds a @volatile annotation to lazy vals. The @volatile annotation is needed to maintain thread-safe behaviour of lazy vals in Dotty.

// before
lazy val x = ...
// after
@volatile lazy val x = ...

With @volatile, Dotty uses a deadlock free scheme that is comparable-if-not-faster than the scheme used in scalac.

ExplicitUnit

Adds an explicit Unit return type to def declarations without a return type

// before
trait A {
  def doSomething
}
// after
trait A {
  def doSomething: Unit
}

Such members already have a return type of Unit and sometimes this is unexpected. Adding an explicit return type makes it more obvious.

DottyVarArgPattern

Replaces @ symbols in VarArg patterns with a colon (:). See http://dotty.epfl.ch/docs/reference/changed/vararg-patterns.html

// before
case List(1, 2, xs @ _*)
// after
case List(1, 2, xs : _*)

NoAutoTupling

Adds explicit tuples around argument lists where auto-tupling is occurring.

To use this rewrite:

// before
def someMethod(t: (Int, String)) = ...
someMethod(1, "something")
// after
def someMethod(t: (Int, String)) = ...
someMethod((1, "something"))

Auto-tupling is a feature that can lead to unexpected results, making code to compile when one would expect a compiler error instead. Adding explicit tuples makes it more obvious.

Note. Some auto-tupling cases are left unfixed, namely the ones involving constructor application using `new`
case class Foo(x: (String, Boolean))
new Foo("string", true) // won't be fixed
Foo("string", true)     // will be fixed
This is a known limitation.

NoValInForComprehension

Removes val from definitions in for-comprehension.

// before
for {
  n <- List(1, 2, 3)
  val inc = n + 1
} yield inc
// after
for {
  n <- List(1, 2, 3)
  inc = n + 1
} yield inc

The two syntaxes are equivalent and the presence of the val keyword has been deprecated since Scala 2.10.

NoExtendsApp

Replaces usage of the scala.App trait with an explicit main function.

// before
object Main extends App {
  println(s"Hello ${args(0)}")
}

// after
object Main {
  def main(args: Array[String]) = {
    println(s"Hello ${args(0)}")
  }
}

The scala.App trait uses DelayedInit, which has been dropped in Dotty. More info here.

Planned rewrites...

See here.

Creating your own rewrite


It is possible to implement custom rewrites with Scalafix. Depending on what your rewrite does, it may be a lot of work or very little work. Don't hestitate to get an estimate on Join the chat at https://gitter.im/scalacenter/scalafix for how complicated it would be to implement your rewrite.

Before you begin

Before you dive right into the code of your rewrite, it might be good to answer the following questions first.

What diff do you want to make?

Scalafix is a tool to automatically produce diffs. Before implementing a rewrite, it's good to manually migrate/refactor a few examples first. Manually refactoring code is helpful to estimate how complicated the rewrite is.

Is the expected output unambiguous?

Does the rewrite require manual intervention or do you always know what output the rewrite should produce? Scalafix currently does not yet support interactive refactoring. However, Scalafix has support for configuration, which makes it possible to leave some choice to the user on how the rewrite should behave.

Who will use your rewrite?

The target audience/users of your rewrite can impact the implementation the rewrite. If you are the only end-user of the rewrite, then you can maybe take shortcuts and worry less about rare corner cases that may be easier to fix manually. If your rewrite is intended to be used by the entire Scala community, then you might want to be more careful with corner cases.

What code will your rewrite fix?

Is your rewrite specific to a particular codebase? Or is the rewrite intended to be used on codebases that you don't have access to? If your rewrite is specific to one codebase, then it's easier to validate if your rewrite is ready. You may not even need tests, since your codebase is your only test. If your rewrite is intended to be used in any random codebase, you may want to have tests and put more effort into handling corner cases. In general, the smaller the target domain of your rewrite, the easier it is to implement a rewrite.

How often will your rewrite run?

Are you writing a one-off migration script or will your rewrite run on every pull request? A rewrite that runs on every pull request should ideally have some unit tests and be documented so that other people can help maintain the rewrite.

scalacenter/scalafix.g8

Run the following commands to generate a skeleton project

cd reponame // The project you want to implement rewrites for.

// --rewrite= should ideally match the GitHub repo name, to make
// it possible for users to run `scalafix "github:org/reponame/v1.0"`
sbt new scalacenter/scalafix.g8 --rewrite="reponame" --version="v1.0"
cd scalafix
sbt tests/test

Note that the scalafix directory is a self-contained sbt build and can be put into the root directory of your repo. The tests are written using scalafix-testkit.

Example rewrites

The Scalafix repository contains several example rewrites and tests, see here. These examples may serve as inspiration for your rewrite.

Vocabulary

The following sections explain useful vocabulary when working with Scalafix.

Rewrite

A rewrite is a small program/function that can produce diffs. To implement a rewrite, you extend the Rewrite class. To run a rewrite, users execute scalafix --rewrites MyRewrite. Multiple rewrites can be composed into a single rewrite. For example, the migration for Dotty may involve ProcedureSyntax, ExplicitUnit, DottyVarArgPattern, ExplicitReturnTypes and a few other rewrites. It is possible to combine all of those rewrites into a single Dotty rewrite so users can run scalafix --rewrites Dotty.

RewriteCtx

A rewrite context contains data structures and utilities to rewrite a single source file. For example, the rewrite context contains the parsed Tree, Tokens, lookup tables for matching parentheses and more.

Patch

A "Patch" is a data structure that describes how to produce a diff. Two patches can combined into a single patch with the + operator. A patch can also be empty. Patches can either be low-level "token patches", that operate on the token level or high-level "tree patches" that operate on parsed abstract syntax tree nodes. The public API for patch operations is available in PatchOps.scala
class SyntacticPatchOps(ctx: RewriteCtx) {
  def removeImportee(importee: Importee): Patch = TreePatch.RemoveImportee(importee)
  def replaceToken(token: Token, toReplace: String): Patch =
    Add(token, "", toReplace, keepTok = false)
  def removeTokens(tokens: Tokens): Patch = PatchOps.removeTokens(tokens)
  def removeToken(token: Token): Patch = Add(token, "", "", keepTok = false)
  def rename(from: Name, to: Name)(implicit fileLine: FileLine): Patch =
    ctx.toks(from).headOption.fold(Patch.empty)(tok => Add(tok, "", to.value, keepTok = false))
  def addRight(tok: Token, toAdd: String): Patch = Add(tok, "", toAdd)
  def addLeft(tok: Token, toAdd: String): Patch = Add(tok, toAdd, "")
}

class SemanticPatchOps(ctx: RewriteCtx, mirror: Mirror) {
  def removeGlobalImport(symbol: Symbol): Patch = RemoveGlobalImport(symbol)
  def addGlobalImport(importer: Importer): Patch = AddGlobalImport(importer)
  def replace(from: Symbol,
              to: Term.Ref,
              additionalImports: List[Importer] = Nil,
              normalized: Boolean = true): Patch =
    Replace(from, to, additionalImports, normalized)
  def renameSymbol(from: Symbol, to: Name, normalize: Boolean = false): Patch =
    RenameSymbol(from, to, normalize)
}
Some things are typically easier to do on the token level and other things are easier to do on the tree level. The Patch API is constantly evolving and we regularly add more utility methods to accomplish common tasks. If you experience that it's difficult to implement something that seems simple then don't hesitate to ask on Join the chat at https://gitter.im/scalacenter/scalafix.

Scalameta

Scalafix uses Scalameta to implement rewrites. Scalameta is a clean-room implementation of a metaprogramming toolkit for Scala. This means it's not necessary to have experience with Scala compiler internals to implement Scalafix rewrites. In fact, Scalafix doesn't even depend on the Scala compiler. Since Scalafix is not tied so a single compiler, this means that Scalafix rewrites in theory can work with any Scala compiler, including Dotty and IntelliJ Scala Plugin.

Scalahost

Scalahost is a compiler plugin for Scala 2.x in the Scalameta project that collects information to build a Mirror. For more information about Scalahost, see the Scalameta documentation.

Token

A token is for example an identifier println, a delimiter [ ), or a whitespace character like space or newline. In the context of Scalafix, a Token means the data structure scala.meta.Token. See Scalameta tutorial for more details. See Wikipedia for a more general definition.

Tokens

Tokens is a list of Token. See Scalameta tutorial

Tree

A Tree is a parsed abstract syntax tree. In the context of Scalafix, a Tree means the data structure scala.meta.Tree. See Scalameta tutorial for more details. See Wikipedia for a more general definition.

Syntactic

A Rewrite is "syntactic" when it does not require information from type-checking such as resolved names (println => scala.Predef.println), types or terms, or inferred implicit arguments. A syntactic rewrite can use Tokens and Tree, but not Mirror.

Semantic

A Rewrite is "semantic" if it requires information from the compiler such as types, symbols and reported compiler messages. A semantic rewrite can use a Mirror.

Mirror

A mirror is a Scalameta concept that encapsulates a compilation context, providing capabilities to perform semantic operations for Semantic rewrites. To learn more about mirrors and its associated concepts (Symbol, Denotation, ...), see the Scalameta tutorial.

scalafix-testkit

Use scalafix-testkit for a rapid edit/run/debug cycle while implementing semantic scalafix rewrites.

Note. You may prefer to use the scalacenter/scalafix.g8 template to generate the following boilerplate. In case of any problems, don't hestitate to ask on Join the chat at https://gitter.im/scalacenter/scalafix.

The following instructions assume you are using sbt. However, scalafix-testkit can be used with any build tool. Please ask on the Gitter channel for help on setting up scalafix-testkit with another build tool.

You will need three projects to run scalafix-testkit.

// project/plugins.sbt
addSbtPlugin("com.eed3si9n" % "sbt-buildinfo" % "0.6.1")
addSbtPlugin("org.scalameta" % "sbt-scalahost" % "1.8.0")
// build.sbt
lazy val testsInput = project.in(file("scalafix/input"))
  .settings(scalametaSourceroot := sourceDirectory.in(Compile).value)
lazy val testsExpectedOutput = project.in(file("scalafix/output"))
lazy val tests = project
  .in(file("scalafix/tests"))
  .settings(
    libraryDependencies += "ch.epfl.scala" % "scalafix-testkit" % "0.4.2" % Test cross CrossVersion.full,
     buildInfoPackage := "myproject.scalafix.tests",
     buildInfoKeys := Seq[BuildInfoKey](
       "inputSourceroot" ->
         sourceDirectory.in(testsInput, Compile).value,
       "outputSourceroot" ->
         sourceDirectory.in(testsExpectedOutput, Compile).value,
       "mirrorClasspath" -> classDirectory.in(testsInput, Compile).value
     )
  )
  .dependsOn(testsInput % Scalameta)
Then glue everything together in a test suite like this.
//scalafix/tests/src/test/scala/SemanticTests.scala
package scalafix.tests

import scala.meta._
import scalafix.testkit._

class SemanticTests
  extends SemanticRewriteSuite(
    Database.load(Classpath(AbsolutePath(BuildInfo.mirrorClasspath))),
    AbsolutePath(BuildInfo.inputSourceroot),
    Seq(
      AbsolutePath(BuildInfo.outputSourceroot)
    )
  ) {
  runAllTests()
}

Specify scalafix configuration inside comment at top of file like this.
/*
rewrites = ExplicitUnit
 */
package test

object ExplicitUnit {
  trait A {
    def x
  }
  abstract class B {
    def x
  }
  trait C {
    def x /* comment */
  }
  trait D {
    def x()
  }
  trait E {
    def x(a: String, b: Boolean)
  }
  trait F {
    def x: String // don't touch this
  }
}
And then testkit checks if rewritten codes match the one in output.
package test

object ExplicitUnit {
  trait A {
    def x: Unit
  }
  abstract class B {
    def x: Unit
  }
  trait C {
    def x: Unit /* comment */
  }
  trait D {
    def x(): Unit
  }
  trait E {
    def x(a: String, b: Boolean): Unit
  }
  trait F {
    def x: String // don't touch this
  }
}

For a full working example, see the scalafix repo.

Note: this working example specifies an extra output directory for dotty. Please check it out. :)

Sharing your rewrite

You have implemented a rewrite, you have tests, it works, and now you want to share it with the world. Congrats! There are several ways to share a rewrite if the rewrite is contained in a single file and uses no external dependencies,

If your rewrite uses a custom library, then it's a bit tricky to share it. See #201 for more updates.

FAQ / Troubleshooting


If you have any questions, don't hesitate to ask on gitter Join the chat at https://gitter.im/scalacenter/scalafix.

Enclosing tree [2873] does not include tree [2872]

Scalafix requires code to compile with the scalac option -Yrangepos. A macro that emits invalid tree positions is usually the cause of compiler errors triggered by -Yrangepos. Other tools like the presentation compiler (ENSIME) or Scalameta Semantic API also require -Yrangepos to work properly.

sbt.Init$RuntimeUndefined: References to undefined settings at runtime.

You might be using an old version of sbt. sbt-scalafix requires sbt 0.13.13 or higher.

Scalafix doesn't do anything

RemoveUnusedImports does not remove unused imports

Make sure that you followed the instructions in RemoveUnusedImports regarding scalac options.

Changelog


0.4.2

0.4.1

First of all, I'd like to welcome Gabriele Petronella, @gabro, to the scalafix team! See #184. Big thanks you everybody who contributed this release via issues, pull requests, online discussions on Join the chat at https://gitter.im/scalacenter/scalafix and and offline discussions at Scaladays Copenhagen last week!

0.4.0

This release represents a significant milestone for Scalafix. Scalafix no longer runs as a compiler plugin as it did in previous releases. Instead, Scalafix runs now independently from the compiler, and uses the Scalameta Semantic API to query information from the compiler.

New features

Breaking changes

git shortlog -sn --no-merges v0.3.0..v0.4.0 tells us that 4 people contributed to this release:

0.3.4

0.3.3

0.3.2

See merged PRs.

0.3.1

0.3.0

0.2.1

0.2.0

0.1.0

Fork me on GitHub