Post

AWS XRAY tracing. Part 3: Sampling & Billing

Posted Updated
Preview Image
By Roman Tsypuk 5 min read

Abstract

AWS x-ray Sampling allows defining traces recording behavior (rate of traces per-second, predicates to filter what traces to record, etc.) Applying different sampling configurations we can set different rates of traces recording on different environments, target specific user/organization/service that requires more/less detailed tracing recording.

And of course by setting the sampling rates we can control xray service billing, because it is related with recorded and scanned traces requests.

Link to AWS XRAY tracing. Part 1: Foundational

Link to AWS XRAY tracing. Part 2: Advanced: Querying & Grouping

Sampling

By utilizing sampling, you can manage the size of traces that are captured and stored, while also providing the flexibility to adjust the sampling behavior without the necessity of redeploying or modifying your code.

The default configuration: record the initial request every second and an additional 5% of the requests beyond that. The reservoir contains 1 res/sec to ensure that the service records at least one trace per second while serving requests.

This is trace sampling monitor from aws console for Default Sampling rule, that almost in realtime shows sampling rates fo traces that are ingested into the system:

img.png

When creating Sampling Rule, we can narrow down the set of traces for which sampling configuration is defined. Matching Criteria can be combined with the following predicates:

  • service name
  • service type
  • host
  • resourceARN
  • HTTPMethod
  • URLPath
  • Annotations

So we can create distinct sampling configurations based on endpoint, particular annotation value present in the trace, host, name, etc. The traces that are following these defined predicates will be recorded based on their own custom defined rates.

Sampling setup from aws cli

AWS cli support CRUD operations with sampling rules:

  • aws xray get-sampling-rules
  • aws xray update-sampling-rule
  • aws xray create-sampling-rule
  • aws xray delete-sampling-rule
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

 aws xray get-sampling-rules
{
    "SamplingRuleRecords": [
        {
            "SamplingRule": {
                "RuleName": "Default",
                "RuleARN": "arn:aws:xray:eu-central-1:xxxxxx:sampling-rule/Default",
                "ResourceARN": "*",
                "Priority": 10000,
                "FixedRate": 0.05,
                "ReservoirSize": 1,
                "ServiceName": "*",
                "ServiceType": "*",
                "Host": "*",
                "HTTPMethod": "*",
                "URLPath": "*",
                "Version": 1,
                "Attributes": {}
            },
            "CreatedAt": 0.0,
            "ModifiedAt": 0.0
        }
    ]
}

Sampling for detailed troubleshooting during Incident Window

One of sampling practical use-cases is a detailed troubleshooting. We can TURN ON particular sampling rule for the short period of time only while we are actively troubleshooting (Incident Window) dedicated user/instance (defined in the predicate of sampling rule) issue to gather traces. After recording is finished we can TURN OFF this sampling rule, since traces are already persisted and ready for offline analysis.

Here, we are creating one Sampling configuration called user_with_error, it includes traces for dedicated user and IP with a fixed rate 100%. So all traces that are following these criteria will be recorded.

We can create up to 25 custom sampling rules in addition to Default rule and observer all of them from aws console. Each of them has own monitor with priority, matching criteria, limit details and Trend graph:

img.png

Sampling configuration in aws lambda function

Also, we can customize sampling rules by including sampling JSON configuration directly inside lambda function.

There are several strategies to instrument lambda sampling behavior:

  • NewCentralizedStrategyWithFilePath will first access xray service to get sampling configuration, if any issue will fall back to locally provided config
  • NewLocalizedStrategyFromFilePath will always use only local (pre-built into lambda) sampling rules configuration.

To update AWS lambda with custom sampling configuration we need to go through following steps:

  • Create sampling rules configuration JSON

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

    {
  "version": 2,
  "rules": [
    {
      "description": "Updated Sampling Rules",
      "host": "*",
      "http_method": "*",
      "url_path": "*",
      "fixed_target": 0,
      "rate": 0.05
    }
  ],
  "default": {
    "fixed_target": 1,
    "rate": 0.1
  }
}

  • Sampling rules configuration file should be included both with binary into aws lambda assembly (if zip command is in use for packaging it should be updated):

    1
2
3

    ...
  zip main.zip main ../../sampling/sampling.json
...

  • Update Lambda’s init function and configure with proper sampling strategy.

    1
2
3
4
5
6
7
8
9

    func init() {
  samplingStrategy, _ := sampling.NewCentralizedStrategyWithFilePath("sampling/sampling.json")
  xray.Configure(xray.Config{
    SamplingStrategy: samplingStrategy,
    ServiceVersion:   "1.2.3",
  })
  newSession := session.Must(session.NewSession())
  ses = newSession
}

Sampling for Billing control

By utilizing sampling rules, we have the ability to specify distinct traces recording settings that correspond to various types of environments. This is particularly important as there is a correlation between amount of recorded traces and billing, which allows for the regulation of costs.

Here is a trace recording calculation for generic application with different sampling rate (SR) configurations and same rate of Requests Per Hour (RPH).

application setup: 20000 RPH, SamplingRate: 10%

Traces Recorded per Month = 20000 requests per hour x 24 hours x 31 days x 10% = 1 488 000 traces

Billable Traces Recorded per Month = 1 488 000 traces - 100,000 traces (free tier) = 1 388 000 traces

Monthly Traces Recorded Charges = 1 388 000 traces * $0.000005 = $6.94

application setup: 20000 RPH, SamplingRate: 1%

Traces Recorded per Month = 20000 requests per hour x 24 hours x 31 days x 1% = 148 800 traces

Billable Traces Recorded per Month = 148 800 traces - 100,000 traces (free tier) = 38 800 traces

Monthly Traces Recorded Charges = 38 800 traces * $0.000005 = $0.192

Overall x-ray Billing

AWS xray billing is combined of Free Tier buffer and on top requests that are billed.

There are 4 trace access patterns that are billed and calculated with different charge and rates:

  • traces recorded
  • traces scanned
  • traces retrieved
  • traces insights (newly added xray feature)

Free Tier

  • The first 100,000 traces recorded each month are free.
  • The first 1,000,000 traces retrieved or scanned each month are free.

Everything on top of Free Tier is billed, pricing is specific to the AWS region, you can check the details and sampled calculator on XRAY pricing

aws, xray, monitoring
aws xray monitoring
This post is licensed under CC BY 4.0 by the author.