Extractor Reference¶

A feature is created by applying a feature expression, which consists of a mix of literal strings and extractors.

For convenience, because a single extractor is by far the most common case, unquoted strings are treated as a single extractor. Consider the extractor ua-req-host. This can be used in the following feature expressions, presuming the host is “example.one”.

Feature String	Extracted Feature
`"Host = {ua-req-host}"`	`Host = example.one`
`"Host = {uareq-host:*<15}"`	`Host = example.one***`
`ua-req-host`	`example.one`
`"ua-req-host"`	`ua-req-host`
`"{ua-req-host}"`	`example.one`
`"{{ua-req-host}}"`	`{ua-req-host}`
`!literal "{ua-req-host}"`	`{ua-req-host}`

Extractors may have or require parameters that affect what is extracted which are supplied as an argument. This enables using these parameters inside a feature expression.

Extractors¶

HTTP Messages¶

There is a lot of information in the HTTP messages handled by Traffic Server and many extractors to access it. These are divided in to families, each one based around one of the basic messages -

User Agent Request - the request sent by the user agent to Traffic Server.
Proxy Request - the request sent by Traffic Server (the proxy) to the upstream.
Upstream Response - the response sent by the upstream to Traffic Server in response to the Proxy Request.
Proxy Response - the response sent by Traffic Server to the user agent.

There is also the “pre-remap” or “pristine” user agent URL. This is a URL only, not a request, and is a copy of the user agent URL just before URL rewriting.

In addition, for the remap hook, there are two other URLs (not requests) that are available. These are the “target” and “replacement” URLs which are the values literally specified in the URL rewrite rule. Note these can be problematic for regular expression remap rules as they are frequently not valid URLs.

Host and Port Handling¶

The host and port for a request or URL require special handling due to vagaries in the HTTP specification. The most important distinction is these can appear in two different places, the URL itself or in the Host field of the request, or both. This can make modifying them in a specific way challenging. The complexity described here is to make it possible to do exactly what is needed. In particular, although the HTTP specification says if the host and port are in the URL and in the Host header, these must be the same, in _practice_ many proxies are configured to make them different before sending the request upstream from the proxy, generally so that the Host field is based on what the user agent sent and the URL is changed to be where the proxy routes the request. Therefore there are extactors which work on the request as a whole, considering both the URL and the Host field, and others that always use the URL.

Beyond this, the port is optional and this presents some problems. One is a result of the ATS plugin API which makes it impossible to distinguish between “http://delain.nl” and “http://delain.nl:80”. Both port 80 and port 443 are treated specially, the former for scheme “HTTP” and the latter for “HTTPS”. I intend to add that at some point but currently it cannot be done. The result is it is difficult to impossible to properly set these values in a configuration language such as TxnBox has and even if possible would be rather painful to do repeatedly. Therefore TxnBox has the concept of “location” which corresponds to the host and port (the HTTP specification calls this the authority but everyone thought using the term was a terrible idea). This makes it easy to access the host, the port, or both. This is more important when interacting with directives to set those values. Here is a chart to illustrate the three terms

url	host	port	loc
http://evil-kow.ex/path	evil-kow.ex	80	evil-kow.ex
http://evil-kow.ex/path:80	evil-kow.ex	80	evil-kow.ex
https://evil-kow.ex/path	evil-kow.ex	443	evil-kow.ex
http://evil-kow.ex:4443/path	evil-kow.ex	4443	evil-kow.ex:4443
https://evil-kow.ex:4443/path	evil-kow.ex	4443	evil-kow.ex:4443

Paths¶

Unfortunately due to how the plugin API works, paths are a bit odd. One result is

Important

Paths do not have a leading slash

Given the URL “http://delain.nl/pix”, the path is “pix”, not “/pix”. The existence of the slash is implied by the existence of the path. There is, unfortunately, no way to distinguish a missing from an empty path. E.g. “http://delain.nl” and “http://delain.nl/” are not distinguishable by looking at the value from a path extractor, both will yield an empty string. This matters less than it appears because both ATS and the upstream will treat them identically. Note this applies only to the slash separating the “authority” / “location” from the path. The path for the URLs “http://delain.nl/pix/charlotte” and “http://delain.nl/pix/charlotte/” are distinguishable.

User Agent Request
Pre-Remap
Rewrite Rule URLs
Proxy Request
Upstream Response
Proxy Response
Transaction
Session
Duration
Utility

User Agent Request¶

ua-req-method¶

Result: string

The user agent request method.

ua-req-url¶

Result: string

The URL in the user agent request.

ua-req-scheme¶

Result: string

The URL scheme in the user agent request.

ua-req-loc¶

Result: string

The location for the request, consisting of the host and the optional port. This is retrieved from the URL if present, otherwise from the Host field.

ua-req-host¶

Result: string

Host for the user agent request. This is retrieved from the URL if present, otherwise from the Host field. This does not include the port.

ua-req-port¶

Result: integer

The port for the user agent request. This is pulled from the URL if present, otherwise from the Host field. If not specified, the canonical default based on the scheme is used.

ua-req-path¶

Result: string

The path of the URL in the user agent request. This does not include a leading slash.

ua-req-query¶

Result: string

The query string for the user agent request if present, an empty string if not.

ua-req-query-value¶

Result: string
Argument: Query parameter key.

The value for a specific query parameter, identified by key. This assumes the standard format for a query string, key / value pairs (joined by ‘=’) separated by ‘&’ or ‘;’. The key comparison is case insensitive. NIL is returned if the key is not found.

ua-req-fragment¶

Result: string

The fragment of the URL in the user agent request if present, an empty string if not.

ua-req-url-host¶

Result: string

Host for the user agent request URL.

ua-req-url-port¶

Result: integer

The port for the user agent request URL.

ua-req-url-loc¶

Result: string

The location for the user agent request URL, consisting of the host and the optional port.

ua-req-field¶

Result: NULL, string, string list
Argument: name

The value of a field in the client request. This requires a field name as a argument. To get the value of the “Host” field the extractor would be “ua-req-field<Host>”. The field name is case insensitive.

If the field is not present, the NULL value is returned. Note this is distinct from the empty string which is returned if the field is present but has no value. If there are duplicate fields then a string list is returned, each element of which corresponds to a field.

Pre-Remap¶

The following extractors extract data from the user agent request URL, but from the URL as it was before URL rewriting (“remapping”). Only the URL is preserved, not any of the fields or the method. These are referred to elsewhere as “pristine” but that is a misnomer. If the user agent request is altered before URL rewriting, that will be reflected in the data from these extractors. These do not necessarily return the URL as it was received by ATS from the user agent. All of these have an alias with “pristine” instead of “pre-remap” for old school operations staff. There are no directives to modify these values, they are read only.

pre-remap-scheme¶

Result: string

The URL scheme in the pre-remap user agent request URL.

pre-remap-url¶

Result: string

The full URL of the pre-remap user agent request.

pre-remap-path¶

Result: string

The URL path in the pre-remap user agent request URL. This does not include a leading slash.

pre-remap-host¶

Result: string

The host in the pre-remap user agent request URL. This does not include the port.

pre-remap-port¶

Result: integer

The port in the pre-remap user agent request URL. If not specified, the canonical default based on the scheme is used.

pre-remap-query¶

Result: string

The query string for the pre-remap user agent request URL.

pre-remap-query-value¶

Result: string
Argument: Query parameter key.

The value for a specific query parameter, identified by key. This assumes the standard format for a query string, key / value pairs (joined by ‘=’) separated by ‘&’ or ‘;’. The key comparison is case insensitive. NIL is returned if the key is not found.

pre-remap-fragment¶

Result: string

The fragment of the URL in the pre-remap user agent request if present, an empty string if not.

Rewrite Rule URLs¶

During URL rewriting there are two additional URLs available, the “target” and the “replacement” URL. These are fixed values from the rule itself, not the user agent. For this reason there are extractors to get data from these URLs but no directives to modify them. These values are available only for the “remap” hook, that is directives invoked from a rule in “remap.config”. Query values are not permitted in these URLs and so no extractor for that is provided.

remap-target-url¶

Result: string

The full target URL.

remap-target-scheme¶

Result: string

The scheme in the target URL.

remap-target-loc¶

Result: string

The network location of the target URL.

remap-target-host¶

Result: string

The host in the target URL. This does not include the port, if any.

remap-target-port¶

Result: integer

The port in the target URL. If not specified, the default based on the scheme is extracted.

remap-target-path¶

Result: string

The path in the target URL.

remap-replacement-url¶

Result: string

The full replacement URL.

remap-replacement-scheme¶

Result: string

The scheme in the replacement URL.

remap-replacement-loc¶

Result: string

The network location in the replacement URL.

remap-replacement-host¶

Result: string

The host in the replacement URL. This does not include the port, if any.

remap-replacement-port¶

Result: integer

The port in the replacement URL. If not specified, the default based on the scheme is extracted.

remap-replacement-path¶

Result: string

The path in the replacement URL.

Proxy Request¶

proxy-req-method¶

Result: string

The proxy request method.

proxy-req-url¶

Result: string

The URL in the request.

proxy-req-scheme¶

Result: string

The URL scheme in the proxy request.

proxy-req-loc¶

Result: string

The network location in the request. This is retrieved from the URL if present, otherwise from the Host field.

proxy-req-host¶

Result: string

Host for the request. This is retrieved from the URL if present, otherwise from the Host field. This does not include the port.

proxy-req-path¶

Result: string

The path of the URL in the request. This does not include a leading slash.

proxy-req-port¶

Result: integer

The port for the request. This is pulled from the URL if present, otherwise from the Host field.

proxy-req-query¶

Result: string

The query string in the proxy request.

proxy-req-query-value¶

Result: string
Argument: Query parameter key.

The value for a specific query parameter, identified by key. This assumes the standard format for a query string, key / value pairs (joined by ‘=’) separated by ‘&’ or ‘;’. The key comparison is case insensitive. NIL is returned if the key is not found.

proxy-req-fragment¶

Result: string

The fragment of the URL in the proxy request if present, an empty string if not.

proxy-req-url-host¶

Result: string

The host in the request URL.

proxy-req-url-port¶

Result: integer

The port in the request URL.

proxy-req-url-loc¶

Result: string

The location in the URL if present, an empty string if not.

proxy-req-field¶

Result: NULL, string, string list
Argument: name

The value of a field. This requires a field name as a argument. To get the value of the “Host” field the extractor would be “proxy-req-field<Host>”. The field name is case insensitive.

If the field is not present, the NULL value is returned. Note this is distinct from the empty string which is returned if the field is present but has no value. If there are duplicate fields then a string list is returned, each element of which corresponds to a field.

Upstream Response¶

upstream-rsp-status¶

Result: integer

The code of the response status.

upstream-rsp-status-reason¶

Result: string

The reason of the response status.

upstream-rsp-field¶

Result: NULL, string, string list
Argument: name

The value of a field. This requires a field name as a argument. The field name is case insensitive.

If the field is not present, the NULL value is returned. Note this is distinct from the empty string which is returned if the field is present but has no value. If there are duplicate fields then a string list is returned, each element of which corresponds to a field.

Proxy Response¶

proxy-rsp-status¶

Result: integer

The code of the response status.

proxy-rsp-status-reason¶

Result: string

The reason of the response status.

proxy-rsp-field¶

Result: NULL, string, string list
Argument: name

The value of a field. This requires a field name as a argument. The field name is case insensitive.

If the field is not present, the NULL value is returned. Note this is distinct from the empty string which is returned if the field is present but has no value. If there are duplicate fields then a string list is returned, each element of which corresponds to a field.

Transaction¶

is-internal¶

Result: boolean

This returns a boolean value, true if the request is an internal request, and false if not.

Session¶

inbound-txn-count¶

Result: integer

The number of transactions, including the current on, that have occurred on the inbound transaction.

inbound-addr-remote¶

Result: IP address

The remote address for the inbound connection. This is also known as the “client address”, the address from which the connection originates.

inbound-addr-local¶

Result: IP address

The local address for the inbound connection, which is the address used accept the inbound session.

inbound-sni¶

Result: string

The SNI name sent on the inbound session.

has-inbound-protocol-prefix¶

Result: boolean
Argument: protocol tag prefix

For the inbound session there is a list of protocol tags that describe the network protocols used for that network connection. This extractor checks the inbound session list to see if it contains a tag that has a specific prefix. The most common use is to determine if the inbound session is TLS

with: has-inbound-protocol-prefix<tls>
select:
-  is-true:
   do: # TLS only stuff.

Note

Checking a request for the scheme “https” is not identical to checking for TLS. Nothing prevents a user agent from sending a scheme at variance with the network protocol stack. This extractor checks the network protocol, not the request.

Checking for IPv6 can be done in a similar way.

with: has-inbound-protocol-prefix<ipv6>
select:
- is-true:
  do: # IPv6 special handling.

inbound-protocol¶

Result: string
Argument: protocol tag prefix

For the inbound session there is a list of protocol tags that describe the network protocols used for that network connection. This extractor searches the inbound session list and if there is a prefix match, returns the matched protocol tag. This can be used to check for different versions of TLS.

with: inbound-protocol<tls>
select:
-  match: "tls/1.3"
   do: # TLS 1.3 only stuff.
-  prefix: "tls"
   do: # Older TLS stuff.
-  otherwise:
   do: # Non-TLS stuff.

inbound-protocol-stack¶

Result: tuple of strings

This extracts the entire stack of tags for the network protocols of the inbound connection as a tuple. This could be used to check for an IPv4 connection

with: inbound-protocol-stack
select:
-  for-any:
      match: "ipv4"
   do:
   # IPv4 only things.

In general, though, has-inbound-protocol-prefix is usually a better choice for doing such checking unless the full stack or a full tag is needed.

inbound-cert-verify-result¶

Result: integer

The result of verifying the inbound remote (client) certificate. Due to issues in the OpenSSL library this can be a bit odd. If the the inbound session is not TLS the result will be X509_V_ERR_INVALID_CALL which as of this writing has the value 69 (:reference). Otherwise, if no client certificate was provided and was not required the result is X509_V_OK which has the value 0. This lack can be detected indirectly by all of the certificate extractors returning empty strings.

inbound-cert-local-issuer-field¶

Result: string
Argument: Entry name.

Extract the value for an entry in the local (server) certificate issuer for an inbound session. This will accept a short or long name as the argument. Note these names are case sensitive.

inbound-cert-local-subject-field¶

Result: string
Argument: Entry name.

Extract the value for an entry in the local (server) certificate subject for an inbound session. This will accept a short or long name as the argument. Note these names are case sensitive.

inbound-cert-remote-issuer-field¶

Result: string
Argument: Entry name.

Extract the value for an entry in the remote (client) certificate issuer for an inbound session. This will accept a short or long name as the argument. Note these names are case sensitive.

If a client certificate wasn’t provided or failed validation, this will yield an empty string.

inbound-cert-remote-subject-field¶

Result: string
Argument: Entry name.

Extract the value for an entry in the remote (client) certificate subject for an inbound session. This will accept a short or long name as the argument. Note these names are case sensitive.

If a client certificate wasn’t provided or failed validation, this will yield an empty string.

outbound-txn-count¶

Result: integer

The number of transactions between the Traffic Server proxy and the origin server from a single session. Any value greater than zero indicates connection reuse.

with: outbound-txn-count
select:
- gt: 10
  do:
   - proxy-rsp-field<Connection>: "close"

Warning

For ATS versions before 10, this will return 0 and the value should not be taken into consideration to determine connection reuse.

Duration¶

A “duration” is a span of time. This is specified by one of a set of extractors.

milliseconds¶

Result: duration
Argument: count

A duration of count milliseconds.

seconds¶

Result: duration
Argument: count

A duration of count seconds.

minutes¶

Result: duration
Argument: count

A duration of count minutes.

hours¶

Result: duration
Argument: count

A duration of count hours.

Utility¶

This is an ecletic collection of extractors that do not depend on transaction or session data.

...¶

Result: any

The feature for the most recent with.

random¶

Result: integer

Generate a random integer in a uniform distribution. The default range is 0..99 because the most common use is for a percentage. This can be changed by adding arguments. A single number argument changes the upper bound. Two arguments changes the range. E.g.

random<199> generates integers in the range 0..199.

random<1,100> generates integers in the range 1..100.

The usual style for using this in a percentage form is
with: random
select:
- lt: 5 # match 5% of the time
  do: # ...
- lt: 25: # match 20% of the time - 25% less the previous 5%
  do: # ...

text-block¶

Result: string
Argument: name

Extract the content of the text block (defined by a text-block-define) for name.

ip-col¶

Argument: Column name or index

This must be used in the context of the modifier ip-space which creates the row context needed to extract the column value for that row. The argument can be the name of the column, if it has a name, or the index. Note index 0 is the IP address range, and data columns start at index 1.

stat¶

Result: integer
Argument: Plugin statistic name.

This extracts the value of a plugin statistic, which is currently limited to integers by Traffic Server.

Note statistic values are eventually consistent, there can be multiple second delays between incrementing a statistic with stat-update and the value changing.

env¶

Result: string
Argument: Variable name

Extract the value of the named variable from the process environment.

inbound-tcp-info¶

Result: integer
Argument: Field name

Extracts a field value from the tcp_info data available on some operating systems. If not available, NULL is returned.

The currently supported fields are

rtt: Round trip time.
rto: Retransmission timeout.
retrans: Retransmits.
snd-cwnd: Outbound congestion window.