WARCannon – catastrophically powerful parallel WARC processing

WARCannon leverages clever use of AWS technologies to horizontally scale to any capacity, minimize cost through spot fleets and same-region data transfer, draw from S3 at incredible speeds (up to 100Gbps per node), parallelize across hundreds of CPU cores, report status via DynamoDB and CloudFront, and store results via S3.

In all, WARCannon can process multiple regular expression patterns across 400TB in a few hours for around $100.

WARCannon requires that you have the following installed:

• awscli (v2)
• terraform (v0.11)
• jq
• jsonnet
• npm (v12 or v14)

ProTip: To keep things clean and distinct from other things you may have in AWS, it’s STRONGLY recommended that you deploy WARCannon in a fresh account. You can create a new account easily from the ‘Organizations’ console in AWS. By ‘STRONGLY recommended’, I mean ‘seriously don’t install this next to other stuff’.

First, clone the repo and copy the example settings.

Grepping the internet isn’t for the faint of heart, but starting with an effective seive is the first start. WARCannon supports this by enabling local verification of regular expressions against real Common Crawl data. First, open lambda_functions/warcannon/matches.js and modify the regex_patterns object to include the regular expressions you wish to use in name: pattern format. Here’s an example from the default search set:

exports.regex_patterns = {
	"access_key_id": /('A|"A)(SIA|KIA|IDA|ROA)[JI][A-Z0-9]{14}[AQ]['"]/g,
};

Sometimes a simple regex pattern isn’t sufficient on its own, and you need some additional steps to ensure you’re returning the right information. In this case, simply adding a function to the exports.custom_functions object with the same key name allows you to perform any additional processing you see fit.

exports.regex_patterns = {
	"access_key_id": /('A|"A)(SIA|KIA|IDA|ROA)[JI][A-Z0-9]{14}[AQ]['"]/g,
};

exports.custom_functions = {
	"access_key_id": function(match) {
		// Ignore matches with 'EXAMPLE' in the text, since this is common for documentation.
		if (match.text(/EXAMPLE/) != null) {
			// Returning a boolean 'false' discards the match.
			return false
		}
	}
}

Note: WARCannon is meant to crunch through text at stupid speeds. While it’s certainly possible to perform any type of operation you’d like, adding high-latency custom functions such as network calls can significantly increase processing time and costs. Network calls could also result in LOTS of calls against a website, which could get you in trouble. Be smart about how you use these functions.

The costs of AWS can be anxiety-inducing, especially when you’re only looking to do some research. WARCannon is built to allow both one-off executions in addition to full campaigns, so you can be confident in the results you’ll get back. Once you’re happy with the results you get with testLocal, you can deploy your updated matches and run a cloud-backed test easily:

warcannon$ ./warcannon deploy
warcannon$ ./warcannon test <warc_path>

This will synchronously execute a Lambda function with the regular expressions you’ve configured, and immediately return the results. This process takes about 2.5 minutes, so don’t be afraid to wait while it does its magic.

Once you’re happy with the results you get in Lambda, you’re ready to grep the internet for real. We’ll first go over some basic housekeeping, then kick it off.

WARCannon uses AWS Simple Queue Service to distribute work to the compute nodes. To ensure that your results aren’t tainted with any prior runs, you can tell WARCannon to empty the queue:

warcannon$ ./warcannon emptyQueue
[+] Cleared [ 15 ] messages from the queue

You can then verify the state of the queue:

warcannon$ ./warcannon status
	Deployed: [ YES ]; SQS Status: [ EMPTY ]
	Job Status: [ INACTIVE ]
	Active job url: https://d201offlnmhkmd.cloudfront.net

Verify the following before proceeding:

1. The SQS Queue is empty
2. The job status is ‘INACTIVE’

In order to create the queue messages that the compute nodes will consume, you must first populate SQS with crawl data. WARCannon has several commands to help with this, starting with the ability to show the available scans. In this case, let’s look at the scans available for the year 2021:

warcannon$ ./warcannon list 2021
CC-MAIN-2021-04
CC-MAIN-2021-10

We have two scans matching the string “2021” to work with. We can now instruct WARCannon to populate the queue based on one of these scans. This time, we need to provide a parameter that uniquely identifies one of the scans. “2021-04” will do the trick. We could choose to populate only a partial scan by also specifying a number of chunks and a chunk size, but we’ll skip that for now.

warcannon$ ./warcannon populate 2021-04
{Created 799 chunks of 100 from 79840 available segments"
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

During deployment, WARCannon automatically provisions a database (warcannon_commoncrawl) and workgroup (warcannon) in Athena that can be used to rapidly query information from CommonCrawl. This can be especially useful for populating sparse campaigns based on certain queries. For example, the following query will search for WARCs that contain responses from ‘example.com’

SELECT
	warc_filename,
	COUNT(url_path) as num
FROM
	warcannon_commoncrawl.ccindex
WHERE
	subset = 'warc'	AND
	url_host_registered_domain IN ('example.com') AND
	crawl = 'CC-MAIN-2021-04'
GROUP BY warc_filename
ORDER BY num DESC

You can use the Athena console to fine-tune your results, but you must run the query from the WARCannon command line if you intend to populate a job with it:

./warcannon queryAthena "SELECT warc_filename, COUNT(url_path) as num FROM warcannon_commoncrawl.ccindex WHERE subset = 'warc' AND url_host_registered_domain IN ('example.com') AND crawl = 'CC-MAIN-2021-04' GROUP BY warc_filename ORDER BY num DESC"

[+] Query Exec Id: 0319486e-1846-491c-badf-2e23ae213974 .. SUCCEEDED

WARCannon can then use the results of a query to populate the queue, and does so based on the warc_filename column from the resultset. As such, it’s recommended that you either group by this column or use distinct() to avoid duplicates. WARCannon will throw an error if this field isn’t present. Populate the queue with Athena results by passing the Query Execution ID to the populateAthena command.

./warcannon populateAthena 0319486e-1846-491c-badf-2e23ae213974

{Created 26 chunks of 10 from 251 available segments"
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

Note: While populating a sparse job for a single domain might seem like a good idea, it often isn’t. The responses from a single domain tend to be spread widely across a large subset of WARCs. This can be seen clearly using the example query above to see that of the ~150,000 records in each WARC, the largest single hit for moderate-sized websites can be in the single-digits.

With the queue populated, we’re ready to fire. WARCannon will do a few sanity checks to ensure everything is in order, then show you the configuration of the campaign and give you one last opportunity to abort before you finalize the order.

warcannon$ ./warcannon fire
[!] This will request [ 6 ] spot instances of type [ m5n.24xlarge, m5dn.24xlarge ]
    lasting for [ 86400 ] seconds.

To change this, edit your settings in settings.json and run ./warcannon deploy
--> Ready to fire? [Yes]:

Pull the trigger by responding with ‘Yes’.

--> Ready to fire? [Yes]: Yes
{
    "SpotFleetRequestId": "sfr-03dd32b8-51f7-4c8e-802b-a702fc3c8c95"
}

[+] Spot fleet request has been sent, and nodes should start coming online within ~5 minutes.
    Monitor node status and progress at https://d201offlnmhkmd.cloudfront.net

The response includes a link to your unique status URL, where you can monitor the progress of your campaign and the performance of each node.

WARCannon results are stored in S3 in JSON format, broken down by each node responsible for producing the results. Athena results are stored in the same bucket under the /athena/ prefix. You can sync the results of a campaign to the ./warcannon/results/ folder on your local machine using the syncResults command.

./warcannon syncResults

sync: s3://warcannon-results-202...
sync: s3://warcannon-results-202...
sync: s3://warcannon-results-202...
...

You can then empty the results buckets with clearResults