Optimizing for Mobile Devices with Varnish Cache

Recently, I wrote about using Varnish Cache to speed up websites. However, not all websites appear identically on all devices. For example, many web applications will deliver different content to mobile devices such as phones, tablets, screen-readers, etc. What happens when Varnish receives a request for a resource from one of these devices?

Without additional configuration, Varnish will return the only version of a resource that it has cached for a particular URL — regardless of its appropriateness for the device performing the request.

This can be problematic. For example, if a mobile phone performs the first request for a resource, the request may return specific mobile content, and Varnish will likely cache it. However, if a desktop browser subsequently performs a request for the same resource, it may receive the mobile content that Varnish has cached. This could cause mobile-specific content to appear in the desktop browser.

Why Varnish Has Mobile Trouble

The situation arises from the mechanism Varnish utilizes to manage cached resources. When Varnish caches a resource, it creates a unique hash from the parameters of the request used to fetch the resource. The hash identifies the resource in the Varnish cache.

When Varnish receives subsequent requests, it hashes select request parameters and attempts to use that hash to match up the request with a resource in the cache. If the hash corresponds with a valid resource, Varnish returns it (a cache hit); if it does not, Varnish fetches it from the back end (a cache miss).

Most web applications utilize the User-Agent header of HTTP requests to determine what content to return: mobile-specific content, or normal desktop content. However Varnish does not, by default, include the User-Agent header as a parameters in hashing requests to identify resources. This explains why Varnish, without additional configuration, returns the same content to different devices, even if the backend web application does not.

Adding User-Agent to Varnish Cache

To fix the problem, Varnish needs to be configured to include the User-Agent header when hashing parameters of a request. However, it would be inappropriate to simply add the entire User-Agent string as a parameter to the hash function. The number of different User-Agents would cause Varnish to cache identical resources separately for each request with a different User-Agent. This would cause the size of the cache to grow dramatically, and decrease performance.

Instead, it would be more appropriate to classify User-Agents, and then cache resources based on this classification.

The following VCL code snippet demonstrates how to classify a request by device type based on User-Agent. The X-Device header stores the device classification for later use.

# Routine to identify and classify a device based on User-Agent
sub identify_device {

  # Default to classification as a PC
  set req.http.X-Device = "pc";

  if (req.http.User-Agent ~ "iPad" ) {
    # The User-Agent indicates it's a iPad - so classify as a tablet
    set req.http.X-Device = "mobile-tablet";
  }

  elsif (req.http.User-Agent ~ "iP(hone|od)" || req.http.User-Agent ~ "Android" ) {
    # The User-Agent indicates it's a iPhone, iPod or Android - so let's classify as a touch/smart phone
    set req.http.X-Device = "mobile-smart";
  }

  elsif (req.http.User-Agent ~ "SymbianOS" || req.http.User-Agent ~ "^BlackBerry" || req.http.User-Agent ~ "^SonyEricsson" || req.http.User-Agent ~ "^Nokia" || req.http.User-Agent ~ "^SAMSUNG" || req.http.User-Agent ~ "^LG")        {
    # The User-Agent indicates that it is some other mobile devices, so let's classify it as such.
    set req.http.X-Device = "mobile-other";
  }
}

Now that the request has been classified based on device, the classification needs to be added to the specialized vcl_hash function that Varnish executes to create the hash used to identify resources in the cache. Recall that VCL operates by modifying default behavior — when defining the vcl_hash function, only the new behavior needs to be specified:

sub vcl_hash {
  # If the device has been classified as any sort of mobile device, include the User-Agent in the hash
  # However, do not do this for any static assets as our web application returns the same ones for every device.
  if (!(req.url ~ ".(gif|jpg|jpeg|swf|flv|mp3|mp4|pdf|ico|png|gz|tgz|bz2)(?.*|)$")) {
    hash_data(req.http.X-Device);
  }
}

Now that requests from devices have been classified, and this classification has been added to the hash function, the identify_device sub-routine must be called when Varnish receives a potentially cacheable request:

sub vcl_recv {

  # Be sure to actually call our sub-routine to classify devices!
  call identify_device;

  if (req.http.Accept-Encoding) {
    if (req.url ~ ".(gif|jpg|jpeg|swf|flv|mp3|mp4|pdf|ico|png|gz|tgz|bz2)(?.*|)$") {
      remove req.http.Accept-Encoding;
    } elsif (req.http.Accept-Encoding ~ "gzip") {
      set req.http.Accept-Encoding = "gzip";
    } elsif (req.http.Accept-Encoding ~ "deflate") {
      set req.http.Accept-Encoding = "deflate";
    } else {
      remove req.http.Accept-Encoding;
    }
  }
  if (req.url ~ ".(gif|jpg|jpeg|swf|css|js|flv|mp3|mp4|pdf|ico|png)(?.*|)$") {
    unset req.http.cookie;
    set req.url = regsub(req.url, "?.*$", "");
  }
  if (req.http.cookie) {
    if (req.http.cookie ~ "(mycookie_|web-app-1-|special-identifier)") {
      return(pass);
    } else {
      unset req.http.cookie;
    }
  }
  set req.grace = 120s;
}

Now Varnish can serve different resources from its cache appropriately to various devices, just as the backend web application intends. Similar configuration changes can be made to cache resources differently for any number of request paramaters.