Dispatch

Dispatch is a library for conducting HTTP interaction. You tell it what to do in pure and uninhibited Scala, and Dispatch carries out your orders using its trusty HttpClient backend.

Version 0.8.10 Release Notes

Try Dispatch

Getting a Console

The simplest way to try Dispatch is to clone the Twine example application from github.

git clone https://github.com/n8han/dispatch-twine.git

If you don’t have git available, you can download and extract the project’s current zip archive.

Twine is built with sbt 0.10. If you haven’t yet setup sbt, now is a good time to do that. Once you have sbt on your executable search path, you can enter its interactive console for the Twine project.

cd dispatch-twine
sbt

In sbt’s console, call the console task.

console

On the first run this task will download Dispatch dependencies and compile the Twine app. After that, you should see a message welcoming you to the console and a friendly scala> prompt.

Your First Request

First, import the main Dispatch classes and objects into scope.

import dispatch._

Then, we’ll need an HTTP executor to carry out our requests.

val h = new Http

Requests are described with dispatch.Request objects, and one way to construct them is with a URL.

val req = url("http://www.scala-lang.org/")

This works because you’ve imported an object called url from the dispatch package, and the object is itself a function that creates request objects.

Now, we have an executor and a request. Dispatch needs to know how to handle the request.

val handler = req >>> System.out

With this complete request-response handler, Dispatch can execute the request. Assuming you have a network connection, that is.

h(handler)

And that was the source of the Scala home page. Of course, we don’t usually assign all these component parts to values unless we need to reuse them. Typically, the handler above would be written in one line.

h(url("http://www.scala-lang.org/") >>> System.out)

If you can’t wait to see more Dispatch verbs like >>>, here’s a colorful cheat sheet.

Choose an Executor

Dispatch supports arbitrarily many types of HTTP executors—you could make your own, and you wouldn’t be the first to do so. Executors carry out your request handlers and callbacks.

Because executors are defined separately from HTTP interaction, you can model HTTP interaction without demanding a particular implementation. For example, the same interface to a web API can be used with an NIO executor and a Google App Engine executor.

Dispatch includes several executors, defined in various modules and packages. These executors are all called “Http”, so application code typically imports only the base dispatch package and refers to executors by their subpackage name, as seen below.

To follow along with the examples below, use a console of the Twine example app described in the preceding page.

Current Thread Blocking (dispatch-http)

The traditional and most common Dispatch executor is located in the base dispatch package. It performs its operations in the current program thread, allowing it to return whatever value the handler builds. If you request a string response body with as_str, the executor’s return type is String.

import dispatch._
val h = new Http
h(url("http://www.scala-lang.org/") as_str)

You may notice in the Scala console that the last expression above reports a return type of HttpPackage. This wrapping type is required to generalize the executor model, but in this executor it is defined as the wrapped type itself. It can be used any place where the wrapped type is expected.

val s: String = h(url("http://www.scala-lang.org/") as_str)

An important caveat of the executor defined above is that it is not thread-safe. You should not share references to it across threads. Instead, Dispatch applications commonly define a function in some convenient object that returns a newly constructed executor. This also allows you replace it with a customized executor, later.

def http = new Http

Another option is to use a thread-safe executor, which maintains a shared connection pool and may (or may not) provide better performance. Dispatch provides a trait for this behavior:

val http = new Http with thread.Safety

The Http singleton object is an executor constructed with this trait, so you can use it from any thread.

Http(url("http://www.scala-lang.org/") as_str)

Background Thread Blocking (dispatch-http)

Another executor that does not depart from the traditional model is the background thread executor.

import dispatch._
val h = new thread.Http
val f = h(url("http://www.scala-lang.org/") as_str)

This executor application evaluates to a future for the string value to be constructed by the handler. A future in Dispatch is a function.

type Future[T] = Function0[T] {
  def isSet: Boolean
}

When applied, the future waits until its underlying value is available, and isSet reports this availability. These are useful when handling a batch of requests, for example.

val urls: Traversable[String] = < some urls I need to get >
val futures = urls.map { u => h(url(u) as_str) }

< do something else that's slow... >

val (ready, working) = futures.partition { _.isSet }
consume(ready.map { _() }) // mapped to Traversable[String]

Native I/O (dispatch-nio)

This executor uses Java NIO and does not block a current or background thread while carrying out requests. You can have n active requests without tying up n threads or waiting for available threads from a pool.

import dispatch._
val h = new nio.Http
val f = h(url("http://www.scala-lang.org/") as_str)

Like the background thread executor, the NIO executor returns a Dispatch future; you can operate on collection as in the previous example. One caveat of this executor is that it must be shutdown.

h.shutdown()

If you’re using this executor in a console and forget to shut it down, the console may not close when asked.

Google App Engine (dispatch-gae)

This executor operates on the current thread, like dispatch.Http, but it is specially modified to work with Google App Engine which does not permit the direct use of sockets.

import dispatch._
val h = new gae.Http
val s = h(url("http://www.scala-lang.org/") as_str)

Note that this executor is not on Twine’s classpath and should only be used with App Engine.

Means of Execution

The examples above all use apply which will execute the handler if the resulting HTTP statusCode is in the range 200 to 204. This range allows the standard handlers to fail-fast in situations where the response wouldn’t be handled. If you need finer control please review the docs for the when and x methods on trait HttpExecutor.

Project Setup

Modules and Artifacts

Dispatch is divided into a number of modules so that client applications need only depend on the parts of Dispatch they use. Some of the modules depend on other modules, and dependencies managers will add these transitive dependencies automatically.

Each module is cross-built against several versions of Scala and published to the scala-tools repository with the organization-id “net.databinder”. The modules have the Scala version they are built against appended. For Scala 2.10.2, the full artifact names are as follows:

Source Dependencies

Because each web API and third party Scala library has its own release timeline, some modules are not part of the main Dispatch build and distribution. Instead, they are external modules hosted on github and tagged for release as needed.

Since these modules are build from source, binary compatibility with Scala and Dispatch versions is not an issue. More importantly, they can be trivially initted, forked, and used by anyone.

If a module is lacking an API method that you need right now, you can fork it, fix it, tag it, and push it without waiting on anyone else. At that point it’s “released” under your name. You should also fork this documentation to add your integration module to the list:

Build Tools

Configuring sbt Projects

When using sbt with Scala binary dependencies, it’s best to have the Scala version automatically appended so it will always match your project’s. In a sbt 0.11.x build.sbt:

libraryDependencies ++= Seq(
  "net.databinder" %% "dispatch-http" % "0.8.10"
)

To use source dependencies with sbt, create a project build such as project/build.scala. This is an example using dispatch-lift-json:

import sbt._
object MyApp extends Build
{
  lazy val root =
    Project("root", file(".")) dependsOn(dispatchLiftJson)
  lazy val dispatchLiftJson =
    uri("git://github.com/dispatch/dispatch-lift-json#0.1.0")
}

The Twine example application uses both source and binary Dispatch dependencies in its sbt project.

Maven

With Maven, you can depend on Dispatch’s binary modules by specifying the full artifact id with Scala version:

<dependency>
  <groupId>net.databinder</groupId>
  <artifactId>dispatch-http_2.10.2</artifactId>
  <version>0.8.10</version>
</dependency>

To use source dependencies with Maven, your best bet is to check out the project as a submodule.

Community

Dispatch (including this documentation) lives on GitHub, and many of its features and bug fixes originate in forks of the project there. Dispatch’s documentation is generated via Pamflet. We also have a mailing list.

A Simple Request

This section explores basic request definition and response handling with Dispatch. You’ll need a Scala console with Dispatch on the classpath to follow along.

URLs and Paths

Hosts are defined with the :/ verb. It should remind you (a little) of the characters that come before hosts in URLs.

import dispatch._
val sl = :/("www.scala-lang.org")

This value sl is a request to the root path of the Scala home domain. What if we want some sub-path of it? There’s an obvious verb for that: /

val learnScala = sl / "node" / "1305"

Now we have a reference to a page (the “Learning Scala” page actually), and sl is unchanged.

Request definitions such as these are immutable; by appending a path to one you are creating a new request object, similar to string concatenation in Java. If we don’t need to use the host for different requests, we would probably have defined this all at once:

val learnScala2 = :/("www.scala-lang.org") / "node" / "1305"

But—can’t you define this with its actual URL? Of course!

val learnScala3 = url("http://www.scala-lang.org/node/1305")

The url verb is also imported from the dispatch package and it produces a request object like the others. It is most useful when dealing with URLs stored externally, or discovered at runtime. For fixed requests in your application code, building up from host and path components is usually preferred as it lends itself to reuse.

Response Bodies

Request definitions, by themselves, are not enough information for Dispatch to do its job. The response that the server returns may have no body at all—or it could contain too much data to hold in memory.

Before Dispatch makes a request, then, you must tell it how to handle the server’s response. For this we have handlers. Handlers are created from request definitions using handler verbs. Taking up the request defined on the last page, we could simply ignore the response.

val ignore = learnScala >|

This would be very unusual for a GET request in particular, which shouldn’t have any side effects. (It’s also not speaking well of our commitment to learning Scala.)

To use a handler, you pass it to a Dispatch executor.

Http(ignore)

Okay, but let’s say you actually want to do something with the response. Assuming it’s text of a reasonable size, you could retrieve it as a string:

Http(learnScala >- { str =>
  str.length
})

Like most handlers this one takes a function from the transformed response body to any type. Here the function is of type String => Int. It merely returns the string length of the response.

How about trying something more interesting with the body, like extracting the page’s title?

import XhtmlParsing._
Http(learnScala </> { nodes =>
  (nodes \\ "title").text
})

The </> handler processes the response as XHTML and passes a scala.xml.NodeSeq to the supplied function. The \\ projection function finds the <title> node, and text gives its contents.

Note: The parser imported here is very brittle. See the next section for alternatives that support real-world HTML.

Parsing Responses

While you can easily wield Dispatch’s stream and reader handlers to bind to any parser you like, several binding modules are available to make it even simpler.

Lift JSON

Lift-JSON is developed and supported by the Lift framework. These are some basic instructions for using it with Dispatch.

Example Usage

Using Lift-JSON is very similar to using Dispatch’s internal JSON representation. As always, start by importing the main Dispatch methods.

import dispatch._

But instead of importing the internal JSON methods, import Dispatch’s interface to lift-json:

import dispatch.liftjson.Js._

We will now be able to use the ># operator in our handlers. For example, if we wanted to search for scala podcasts, we might use something like

import net.liftweb.json.JsonAST._

val http = new Http()
val u = url("http://gpodder.net/search.json") <<? Map("q" -> "scala")
http(u ># { json => 
  (json \ "title" children) flatMap( _ match {
    case JField("title", JString(d)) => Some(d)
    case JString(d) => Some(d)
    case _ => None
  })
})

This script starts by importing lift’s JSON values (for pattern matching) and then creates the http executor and the gpodder url. We then execute with a handler which accepts a lift JValue and returns a list of the podcast titles. Simple, no?

TagSoup

TagSoup is a SAX-compliant parser, that parses HTML instead of well formed XML. TagSoup is very resilient, allowing you to load HTML as found in the wild, to be processed as a scala NodeSeq.

Example Usage

To process the response with TagSoup you first have to make the handler verbs available. This can be done in several ways.

import dispatch.tagsoup.TagSoupHttp._

Now we can use the operators </>, tagsouped and as_tagsouped in our handlers. If we want to find the title of a HTML page it can look something like this.

import dispatch.tagsoup.TagSoupHttp._

val title = Http(:/("example.org") </> { ns =>
              (ns \ "title").text
            })

TagSoup let’s you work with the HTML as a scala.xml.NodeSeq and as a convenience you can use as_tagsouped to retrieve it.

val ns = Http(:/("example.com") as_tagsouped)

JSoup

JSoup is a library for working with real-world HTML. It parses HTML and provides a convenient API for extracting and manipulating data. JSoup is similar to TagSoup as they both lets you work with real-world HTML, but JSoup provide more functionality for working with the extracted result.

Example Usage

To use JSoup you have to make it handler verbs available. This is easiest done by importing them from JSoupHttp.

import dispatch.jsoup.JSoupHttp._

Now we can use the operators </>, jsouped, as_jsouped and as_jsoupedNodeSeq in our handlers. As a start we can extract the title from a HTML page.

import dispatch.jsoup.JSoupHttp._

val title = Http(:/("example.org") </> { doc =>
              doc.title
            })

JSoup parse the HTML to a DOM like structure org.jsoup.nodes.Document. There is a rich set of find and search methods returning org.jsoup.nodes.Elements. Elements implements java List so they are very easy to use with scala collections, just make the usual import scala.collection.JavaConversions._.

So, to extract all links from a page and put them in a list as absolute paths looks like this:

import dispatch.jsoup.JSoupHttp._
import scala.collection.JavaConversions._

val request = :/("example.org") / "test.html"
val list = Http(request </> { doc =>
  doc.select("a[href]").map(_.attr("abs:href"))
})

JSoup is a great api for processing HTML, and scala makes it even better. To learn more of it’s capabilities take a look in the JSoup Cookbook.

Putting It All Together

Once you’ve mastered the basic request and response verbs, you can compose almost any kind of HTTP interaction.

Two Handlers Are Better Than One

Dispatch has so many handy response handlers—but what if you want to use two with the same request? You can’t do that with a stream of the response body, since this can only be consumed once, but for headers it’s essential and straightforward.

Header Chaining

The recommended header-handling verb >:+ provides a header Map and a request object to chain a second handler for the body.

http(:/("dispatch.databinder.net") >:+ { (headers, req) =>
  headers("content-type").filter {
    _.contains("text/html")
  }.headOption.map { _ =>
    req </> { nodes => (nodes \\ "h1").text }
  }.getOrElse {
    req >> { _ => "unknown content type" }
  }
})

To facilitate case-insensitive handling, the keys in the header Map are uniformly lowercase.

The function passed to >:+ should produce a second handler for the body of the response. In this case, it produces different response handlers depending on the headers present.

Dispatch in Android

Many people are writing Android applications in Scala and want to do their HTTP communications with Dispatch. Unfortunately, this is not as easy as it should be.

Android’s HttpClient

The Android platform ships with the Apache HttpClient library. It is packaged in the same JAR as the “Java” standard library. From an application perspective it is just there, always.

When packaging for Android, you should be sure not to include any HttpClient jars; the classloader will always find the stock client’s classes first. This allows apps that require HTTP communication—nearly all apps—to save space.

The downside of shipping libraries in the standard classpath is that the environment becomes rigidly coupled to them. If you want to take advantage of a new feature in a library, too bad. You’d have to wait until all of your users upgrade their operating system to the new version—if it’s ever made available for their devices.

But wait, it gets worse! Updates to the HttpClient libraries in Android stalled. There was a procedural issue with binary compatibility; see this thread for the gory details. The short of it is, we can rule out just waiting for those parties to fix the situation.

Dealing with It

Versions of Dispatch through 0.7.x can work with Android’s random old version of HttpClient just fine. Version 0.8.x however depends on newer major versions of the HttpComponents libraries and certain interfaces that are not compatible with the code shipped with Android.

It is possible to include the newer HttpComponents artifacts in the application package and to trick the classloader into finding them. How? You can obfuscate the class and package names with ProGuard, eliminating the namespace collisions.

But really: isn’t programming hard enough without some bytecode analyzer changing the names of all your symbols prior to runtime? For most of us, yes. That’s why it’s recommended here that you use Dispatch 0.7.x with Android. That works without any tricks and it’s still a lot nicer than having to code to the client directly.

What’s in Store

This unfortunate problem underscores the need for Dispatch to be independent of any single backend. The primary advancement of 0.8 was to support HttpComponet’s async NIO client in addition to its standard blocking client, but that (like the rest of 0.8) is unusable on Android without some serious hackery.

Dispatch 0.9 will therefore need to support entirely different backends such as Netty or one of the higher level async clients built on Netty. And main reason to choose a backend like that for Andorid apps will be, ironically enough, its absence in the platform distribution.

API Reference

Dispatch Periodic Table

This clever and attractive table lists Dispatch’s request and response verbs all in one place.

Scaladocs

Generated documentation is available for the latest version of Dispatch. Since most Dispatch interaction is defined by verbs that are added implicitly to requests, the interesting classes are RequestVerbs, CallbackVerbs, and so on.

Who’s Using Dispatch?

These are some projects and companies that have used Dispatch. Are you using Dispatch? Fork this page on github.

Novus

Dispatch forms the foundation of an in-house SOAP client at Novus together with scalaxb. Thanks to these tools our standard response to “Can you do SOAP?” is “Bring it on!” instead of “Ew, what’s this on my shoe?!”

MLB.com

In a past life, maxaf wrote an internal baseball stats REST API client that was used to feed data into some very grueling PDF generation code. Dispatch was used to cleanly produce HTTP requests & parse responses without hurting engineer morale.

conscript

conscript uses Dispatch to fetch program descriptors from github.

giter8

giter8 uses Dispatch to fetch templates from github.

kiln2rally

kiln2rally uses Dispatch along with lift-json to hit Rally’s awesome web API.

posterous-sbt

The posterous-sbt plugin uses Dispatch to publish project release notes.

Unfiltered

Unfiltered uses Dispatch in its system tests.

Fork me on GitHub