It takes standard headers and knowledge about objects in the cache and
potentially rewrites those headers so that a client will use a URL
that’s already cached instead of one that isn’t. The headers are
specified in RFC 6249 (Metalink/HTTP: Mirrors and Hashes) and RFC 3230
(Instance Digests in HTTP) and are sent by various download
redirectors or content distribution networks.
A lot of download sites distribute the same files from many different
mirrors and users don’t know which mirrors are already cached. These
sites often present users with a simple download button, but the
button doesn’t predictably access the same mirror, or a mirror that’s
already cached. To users it seems like the download works sometimes
(takes seconds) and not others (takes hours), which is frustrating.
An extreme example of this happens when users share a limited,
possibly unreliable internet connection, as is common in parts of
Africa for example.
How it Works
When the plugin sees a response with a
header and a header, it checks if
the URL in the header is already cached. If it
isn’t, then it tries to find a URL that is cached to use instead. It
looks in the cache for some object that matches the digest in the
header and if it succeeds, then it rewrites the
header with that object’s URL.
This way a client should get sent to a URL that’s already cached and
won’t download the file again.
The plugin implements the
hook to check and potentially rewrite the and
headers after responses are cached. It doesn’t
do it before they’re cached because the contents of the cache can
change after responses are cached. It uses
check if the URL in the header is already
cached. In future, the plugin should also check if the URL is fresh
The plugin implements the
hook and a null transformation
to compute the SHA-256 digest for
content as it’s added to the cache. It uses SHA256_Init(),
SHA256_Update(), and SHA256_Final() from OpenSSL to compute the
digest, then it uses
TSCacheWrite() to associate the digest
with the request URL. This adds a new cache object where the key is
the digest and the object is the request URL.
To check if the cache already contains content that matches a digest,
the plugin must call
TSCacheRead() with the digest as the key,
read the URL stored in the resultant object, and then call
TSCacheRead() again with this URL as the key. This is
probably inefficient and should be improved.
An early version of the plugin scanned headers. If the URL in the
header wasn’t already cached, it scanned headers for a URL that was. The header is superior because it will find content that
already exists in the cache in every case that a header would, plus in cases where the URL is not
listed among the headers,
maybe because the content was downloaded from a URL not participating
in the content distribution network, or maybe because there are too
many mirrors to list in
The header is also more efficient
than headers because it
involves a constant number of cache lookups. RFC 6249 requires a
header or headers MUST be ignored:
If Instance Digests are not provided by the Metalink servers, the
header fields pertaining to this specification
MUST be ignored.
Metalinks contain whole file hashes as described in Section 6, and
MUST include SHA-256, as specified in [FIPS-180-3].