code

We just noticed that on your first visit to http://snapsort.com, all of the links on the page have the java ;JSESSIONID =foo session id on them. In fact, google is indexing some of our pages with these ids in them, and then complaining that we have some pages with duplicate titles!

We're not the only ones who have faced this problem.  Here is a link to a write up claiming that jsessionid is considered harmful, with instructions on how to remove them using a servlet filter. And here's a link to a search on google for pages indexed with jsessionid, I'm seeing 736 million :)

These JSESSIONIDs are meant to be used to support sessions with users who do not have cookies.  Right now we don't have any session state, and even if we did we'd probably live if users without cookies got no session, given they are probably in the minority, and for our site I imagine session features would be little nice-to-haves such as remembering previous camera comparisons, not the end of the world if you don't get those, compared to say a shopping cart at amazon.

So, we're going to get rid of the pesky JESSIONIDs.  JSESSIONIDs begone!

1. We've added a rewrite rule to NGINX so that if anyone (say, google) comes to our site with a url with a JSESSIONID in it, we'll send them back a 301 permanent redirect response pointing them to a url with the jsessionid stripped.  This will clean up the urls that are already out there, we hope.
2. We've instructed Jetty to not use URL sessions. This will prevent any more urls with jsessionids in them from getting out.

See general Jetty info on configuring session support

Here is the maven XML to configure Jetty to disable JSESSIONIDs:

<webAppConfig implementation="org.mortbay.jetty.plugin.Jetty6PluginWebAppContext">
  <sessionHandler implementation="org.mortbay.jetty.servlet.SessionHandler">
    <sessionManager implementation="org.mortbay.jetty.servlet.HashSessionManager">
      <!-- Disable url sessions using JSessionID -->
      <sessionURL>none</sessionURL>
    </sessionManager>
  </sessionHandler>
</webAppConfig>

And here is how to do it in Scala code (you can probably translate it to Java easily too).  Note: you only need one of these, I used both to support different ways of launching Jetty.

  val server = new Server(8080)
  val context = new WebAppContext()
  context.setServer(server)
  context.setContextPath("/")
  // Disable JSessionIDs!  this will break cookie-less sessions, we'll live!
  context.getSessionHandler.getSessionManager.setSessionURL("none")
  context.setWar("src/main/webapp")

(Update: my previously posted solution opened up a bit of security hole with the catch-all sitemap menu item, so I've revised this solution based on advice from David Pollak)

If you're building a website, its nice to have a custom 404 page.  That way, if users type in an incorrect url, or, they follow an old link, you can show them a friendly page and hopefully give them something to help find what they might be looking for.

Stackoverflow has a great 404 page, for example, try browsing to: http://stackoverflow.com/foobar

Notice two important details:

• The url in your browser is still the incorrect one you click on, this lets you see what page you tried to go to, and possibly fix it
• Using Firebug or something similar, you can see that the status code for this page is 404, not 200 OK.  This helps search engines determine that this page is no longer there.

I'd found articles on the web indicating how to add a custom 404 handler to Lift, but they involved a redirect, which breaks both the details above: you get sent to a new URL, the client receives a 200 status code instead of a 404.  This has been called the soft 404 issue.  I highly recommend you do return a 404 status code so that search engines are not confused, and that you don't redirect the user so they can see the incorrect url.  See my post on my other blog about developing applications that are of the internet.

Here's a solution that works with Lift 1.1-M8.

First, a method to generate the page to show to users when the specified url cannot be found, it is merely some boiler plate code to handle rendering a standard lift template page that you've designed, in this case called 404.html:

  def generate404 = {
    import scala.xml.Node
    
    val resp: Box[Node] =  S.setVars("expandAll" -> "true")  {
      for {
        rendered <- S.runTemplate("404" :: Nil)
      } yield mergeToHtmlHead(rendered)(0)
    }

    XhtmlResponse(resp openOr <html><body>Got a 404</body></html>,
                  Full(DocType.xhtmlStrict), List("Content-Type" -> "text/html; charset=utf-8"),Nil, 404, S.ieMode)     
  }

Next, add this code to your Boot.scala file:

// Catch 404s 
 LiftRules.passNotFoundToChain = false 
 LiftRules.uriNotFound.prepend { 
       case (req, _) => XhtmlTemplateResponse(notFoundNode, 404) 
 }

Next, create a template to show users when when Lift cannot find a template or handler for a given Url, e.g. 404.html:

<lift:surround with="default" at="content">
    <head>
        <title>Page Not Found </title>
    </head>
        <h1>Page Not Found</h1>
        <p>We couldn't find the page you were looking for.</p>
</lift:surround>

Finally, the mergeToHead method is not currently exposed by Lift, so as a work-around I've copied its source into my Boot class. Obviously this is a horrible solution, but it works well for now.  It sounds like this method will be exposed in future Lift versions, and/or there might even be a more elegant solution available.

If your 404 template doesn't require head merging (e.g. it doesn't have a <head> section in the HTML to specify a title) then you won't need this.

   /**
   * TODO: I copied this out of http://scala-tools.org/scaladocs/liftweb/1.0/net/liftweb/util/HeadHelper.scala.html
   * 
   * DPP says we won't need to do this in a future release of Lift
   * 
   * This method finds all <head> tags that are descendants of
   * <body> tags in the specified NodeSequence and merges
   * the contents of those tags into the <head> tag closest
   * to the root of the XML tree.
   */
  def mergeToHtmlHead(xhtml: NodeSeq) : NodeSeq = {
    def trimText(in: NodeSeq): NodeSeq = in flatMap {
      case e: Elem =>
        Elem(e.prefix, e.label, e.attributes, e.scope, trimText(e.child) :_*)

      case g: Group =>
        trimText(g.child)

      case t: Text =>
        val s = t.text.trim
        if (s.length == 0) NodeSeq.Empty
        Text(s)

      case x => x
    }

    val headInBody: NodeSeq =
    (for (body <- xhtml \ "body";
          head <- findElems(body)(_.label == "head")) yield trimText(head.child)).
    toList.removeDuplicates.flatMap(a => a)

    if (headInBody.isEmpty) xhtml
    else {
      def xform(in: NodeSeq, inBody: Boolean): NodeSeq = in flatMap {
        case e: Elem if !inBody && e.label == "body" =>
          Elem(e.prefix, e.label, e.attributes, e.scope, xform(e.child, true) :_*)

        case e: Elem if inBody && e.label == "head" => NodeSeq.Empty

        case e: Elem if e.label == "head" =>
          Elem(e.prefix, e.label, e.attributes,
               e.scope, e.child ++ headInBody :_*)

        case e: Elem =>
          Elem(e.prefix, e.label, e.attributes, e.scope, xform(e.child, inBody) :_*)

        case g: Group =>
          xform(g.child, inBody)

        case x => x
      }

      xform(xhtml, false)
    }
  }

I'm working with a number of XML files right now, and decided to use XInclude to enable one XML file to reference (and include) another one.

Here is an example. First, here is employee.xml:

<?xml version="1.0" encoding="UTF-8"?>
<employeeInfo xmlns:xi="http://www.w3.org/2001/XInclude">
    <xi:include href="common.xml" xpointer="element(/1/1)"/>
    <name>John Doe</name>
</employeeInfo>

Which you can see uses the xi:include element to reference the following file common.xml:

<?xml version="1.0" encoding="UTF-8"?>
 <commonEmployeeInfo>
    <company>Acme</company>
 </commonEmployeeInfo>

We can then use xmllint to process the include for us:

xmllint --xinclude employee.xml

Which produces:

<?xml version="1.0" encoding="UTF-8"?>
<employeeInfo xmlns:xi="http://www.w3.org/2001/XInclude">
    <company>Acme</company>
    <name>John Doe</name>
</employeeInfo>

So, now that we've got our employee.xml file setup, and referencing the common xml file, I wanted to write Scala code to load in the employee.xml file and process the XInclude element like xmllint did for us.

By default XML.Load doesn't process XInclude statements, here is how to do it:

import scala.xml._
import scala.xml.parsing._
import scala.xml.include._
import org.xml.sax.InputSource

import javax.xml.parsers.SAXParser
import javax.xml.parsers.SAXParserFactory
import javax.xml.validation.Schema
import javax.xml.validation.ValidatorHandler
import org.xml.sax.XMLReader


/**
  * A class which makes it possible to turn on XInclude processing of XML files
  * 
  * E.g. one XML file can include another
  * 
  * Example usage:
  * 
  *   val is = new InputSource(new java.io.FileReader("../ontology/DigitalCamerasOntology.xml"))
  *   is.setSystemId("../ontology/")   
  *   val xml = new XIncludeAwareFactoryAdapter().loadXML(is)
  *
  */
class XIncludeAwareFactoryAdapter extends NoBindingFactoryAdapter {
  override def loadXML(source: InputSource) = {
    // create parser
    val f = SAXParserFactory.newInstance
    f.setNamespaceAware(true) // needed otherwise our xincludes don't seem to get processed
    f.setXIncludeAware(true)
    val parser = f.newSAXParser
    
    val xr = parser.getXMLReader
    xr.setContentHandler(this) // needed otherwise rootElem is null

    // parse file
    scopeStack.push(TopScope)
    xr.parse(source)
    scopeStack.pop
    rootElem.asInstanceOf[Elem]
  }
  
  def loadXMLFromFile( filename: String ) = {
    val inputSource = new InputSource(new java.io.FileReader(filename))
    // Find the path of the file so we can process XIncludes
    val path = new java.io.File(filename).getPath
    inputSource.setSystemId(path)
    loadXML(inputSource)    
  }
}

You can then use it like this:

val xml = new XIncludeAwareFactoryAdapter().loadXMLFromFile(filename)

I started working with XStream as a tool to quickly generate XML and/or JSON from Scala objects, it looks very powerful, but unfortunately needs some tweaking to produce simple and clean XML.

This code:

println(new XStream.toXML(List(1,2,3)))

produces this XML:

<scala.coloncolon serialization="custom">
  <unserializable-parents/>
  <scala.coloncolon>
    <int>1</int>
    <int>2</int>
    <int>3</int>
    <scala.ListSerializeEnd/>
  </scala.coloncolon>
</scala.coloncolon>

Instead I wanted something like this:

<list>
  <int>1</int>
  <int>2</int>
  <int>3</int>
</list>

Turns out this is possible, by writing your own custom converter:

Here's the code:

import com.thoughtworks.xstream.converters._
import com.thoughtworks.xstream.converters.collections._
import com.thoughtworks.xstream._
import com.thoughtworks.xstream.mapper._
import com.thoughtworks.xstream.io._

class ListConverter( _mapper : Mapper )  extends AbstractCollectionConverter(_mapper) {
  def canConvert( clazz: Class[_]) = {       
    // "::" is the name of the list class, also handle nil
    classOf[::[_]] == clazz || classOf[scala.Nil$] == clazz
  }
  
  def marshal( value: Any, writer: HierarchicalStreamWriter, context: MarshallingContext) = {
    val list = value.asInstanceOf[List[_]]
    for ( item <- list ) {      
      writeItem(item, context, writer)
    }
  }
  
  def unmarshal( reader: HierarchicalStreamReader, context: UnmarshallingContext ) = {
    var list : List[_] = Nil 
    while (reader.hasMoreChildren()) {
      reader.moveDown();
      val item = readItem(reader, context, list);
      list = list ::: List(item) // be sure to build the list in the same order
      reader.moveUp();
    }
    list
  }
}

object ListConverter {
  def configureXStream( stream: XStream ) = {
    stream.alias("list", classOf[::[_]])
    stream.alias("list", classOf[scala.Nil$])
    stream.registerConverter( new ListConverter(stream.getMapper) )        
  }
}

Scala's library includes a class called Option which can be used to represent an optional value, instead of using null, it has two states: None and Some[T], so either it has a value, or it doesn't.

I needed a class to represent three states (tristate): Unknown (you don't know the value), Known.Some (you know the value), Known.None (you know there is no value).  An example of where you'd use this might be for a property like radioType of class car.  You might use a tristate value to indicate that you don't know the type of the radio on this car, or that you do know the type of radio, or that there is no radio.

Code:

/*
 * Similar to Option[T], but represents three states: known-some, known-none, and unknown
 */
sealed abstract class KnownUnknown[+T] extends Product {
  def isEmpty : Boolean
  def get : T
  
  def isDefined : Boolean = !isEmpty
}

/*
 * Represents the unknown state
 */
case object Unknown extends KnownUnknown[ Nothing ] {
  def isEmpty = true
  def get = throw new NoSuchElementException("Unknown.get")
}

/*
 * Represents the known state (either you know its Some(T) or its None)
 */
final case class Known[+T](_value : Option[T]) extends KnownUnknown[T] {
  if ( _value == null )
    throw new NullPointerException
  
  def this(value : T) = this(Some(value))
  
  def isEmpty = false
  def get = _value.get
}

And a few basic tests:

class KnownUnknownTest extends TestCase("KnownUnknown") {
  def testSome = {
    var value : KnownUnknown[Int] = null
    value = new Known(5)
    
    value match {
      case Known(known) =>
        known match {
          case Some(number) => assertEquals(5, number)
          case None => fail()
        }
      case Unknown => fail()
    }
    
    assertEquals(5, value.get)
  }
  
  def testNone = {
    var value : KnownUnknown[Int] = null
    value = new Known(None)
    
    value match {
      case Known(known) =>
        known match {
          case Some(number) => fail()
          case None =>  // good
        }
      case Unknown => fail()
    }
    
    try {
      value.get
    }
    catch {
      case e:NoSuchElementException => // good
      case _ => fail()
    }    
    
    assert(true) // so we get picked up as a test
  }

  def testUnknown = {
    var value : KnownUnknown[Int] = null
    value = Unknown
    
    value match {
      case Known(known) => fail()
      case Unknown => // good
    }
    
    try {
      value.get
    }
    catch {
      case e:NoSuchElementException => // good
      case _ => fail()
    }
    
    assert(true) // so we get picked up as a test
  }
}

In my past I've done a lot of development with C#, and loved the new language features introduced in 3.5 such as LINQ, extension methods, lambdas etc.

Here's how you can implement extension methods (in C# terminology) in Scala using Implicits.  The goal in this example is to recreate toDictionary that exists in C#, and allow us to take a list of things and put them into a Dictionary (or HashMap in Scala) in one line.

package org.acme

import scala.collection.mutable.HashMap

class IterableExtensions[A,B](pairs : Iterable[(A,B)] ) {
   def toHashMap() : HashMap[A,B] = {
      val hashMap = new HashMap[A,B]
      hashMap ++= pairs
      hashMap
   }
}

object IterableExtensions {
    implicit def IterableExtensions[A,B](i : Iterable[(A,B)]) = new IterableExtensions(i)
}

(Check out this good introduction to implicits in Scala).   Basically we're defining a method toHashMap, and putting it in the singleton IterableExtensions which we can later import, and use like this:

package org.acme

import scala.collection.mutable.HashMap
import org.acme.IterableExtensions._


class Person( _name : String, _age : Int ) {
   def getName = _name
   def getAge = _age
}


class Test {
   // Compose a list of pairs in the form (name, age)
   val peopleWithAge = List ( new Person("John",25), new Person("Bob", 30), new Person("Frank", 35))

   // Put those pairs into a HashMap
   val peopleByAge = peopleWithAge.map( p => p.getAge ).toHashMap
}

In the above code we've imported the IterableExtensions, and they are now accessible off the Iterable interface, which is most (or all?) collections and lists.

C# first introduced me to the world of functional programming, which has led me to Scala.  One feature of C# that I used a lot is the using statement, which lets you write something like this:

using( var connection = new Connection(...)  ) {
  connection.DoStuff();
}

instead of:

Connection connection = null;

try {
   connection = new Connection( ... )
   
   connection.DoStuff();
}
finally {
  connection.Dispose();
}

The first snippet of code is smaller and more concise.  The using statement means you don't have to write boiler plate try-finally code to close your resources in the event of an exception.

Having just started developing in Scala, I was looking for something similar, here is an implementation of Using in Scala:

(Inspired by Beginning Scala by David Pollak creator of the Scala web framework Lift)

object Using {
  /*
   * Recreate C#'s Using statement
   * 
   * Use a method called apply, which in Scala can be invoked directly off the object, e.g.
   * Using(foo) instead of Using.apply(foo)
   */
  def apply[A <: {def close(): Unit}, B](param: A)(f: A => B) : B =
    try {
      f(param)
    } finally {
      param.close()
    }
}

Which can be used like this:

using( new Connection(...)  ) { connection =>
   connection.DoStuff();
 }

So there are a few things going on here:

• "object Using" is the syntax for a singleton class named Using
• By naming our method "apply" we can take advantage of some Scala syntactic sugar to enable us to invoke it like "Using( ... )" instead of the traditional "Using.apply( ... )"
• The method apply takes two parameter lists, the first (param: A) has one parameter, the item to be closed, and the second (f: A => B) which takes a function which will be invoked in a try-finally clause in case it throws an exception