Hanzo

A key market for Hanzo is litigation, where law firms need to able to prove what was or wasn't said by a company or individual at any point in time. A daily capture is a good way to capture this data, but identifying changes would be a long and manual process.

The change detection feature was born out of an R&D project I lead for Hanzo where we'd collect various metrics and try to identify patterns or changes between them. I wrote an initial POC for this mostly using JavaScript and processing a sample of WARCs where we knew there were changes over time.

The key takeaways from the exercise were that we could get quite a long way with word counts, hashing and page counts; while there'd still be a lot of noise, which tends to be the norm in web capture, we could demonstrably detect changes between captures, enough so that it'd be easier for the end-user to find what they were looking for.

After the change detection project we were able to identify changes between captures, but we needed to be able to display that to the end user. The initial brief was to perform image differentials, which I did try, but had pretty low quality results.

Hanzo already had playback tech that made replaying a capture possible both via URL rewriting and via their custom browser with high precision, streaming the data from S3; it was faster than you'd think, real fast actually. I decided to build out the compare tool by streaming this data through htmldiff before serving it to the user, allowing me to offer up both split and unified differentials. The result was really cool, and working in more cases than you'd think, basically everywhere except certain JavaScript heavy pages like social media would produce a useful differential.

Hanzo populate a Solr index as part of their crawl strategy, we integrated this in a way that would allow users to search across time with pages grouped by URL and ungrouped. Text extraction and tagging meant that finding the captured content you're looking for worked fairly well.

Web archiving is complicated to say the least; Hanzo have a whole team of crawl engineers who specialise in the art of performing a successful web capture, there's lots of knobs and levers which for the most part lived in command line arguments until the web application was born.

Request a capture is a project Hanzo drafted to put these powers in the hands of their end users. Working closely with their CTO and Development Manager I helped them crystallise the options that were available and how we'd present them to end-users in a way that they'd understand.

The result was a seemingly simple form with a lot of power. Users specify the capture type, be that a standard website, or a Facebook or Twitter page, and that would govern which crawl driver is used, and also be able to supply settings for how many hops to go off-domain, or whether to include or exclude certain URLs.