Application Tracing Skill

Overview

Distributed traces in Dynatrace consist of spans - building blocks representing units of work. With Traces in Grail, every span is accessible via DQL with full-text searchability on all attributes. This skill covers trace fundamentals, common analysis patterns, and span-type specific queries.

---

Use Cases

1. Investigate Slow Requests

• Goal: Find and diagnose requests exceeding a latency threshold
• Trigger: “slow requests”, “high latency”, “p99 response time”, “find traces over 5 seconds”
• Done: List of slow traces with duration, endpoint, service, and trace IDs for drilldown

2. Analyze Request Failures

• Goal: Identify failed requests, failure reasons, and exception patterns
• Trigger: “failed spans”, “HTTP 500 errors”, “exception analysis”, “failure rate by service”
• Done: Failure breakdown by reason (HTTP code, exception, gRPC status) with exemplar traces

3. Map Service Dependencies

• Goal: Understand service-to-service communication patterns and external API calls
• Trigger: “service dependencies”, “what services does X call”, “outgoing HTTP calls”
• Done: Dependency map showing call counts, latency, and error rates between services

---

Core Concepts

Understanding Traces and Spans

Spans represent logical units of work in distributed traces:

• HTTP requests, RPC calls, database operations
• Messaging system interactions
• Internal function invocations
• Custom instrumentation points

Span kinds:

• span.kind: server - Incoming call to a service
• span.kind: client - Outgoing call from a service
• span.kind: consumer - Incoming message consumption call to a service
• span.kind: producer - Outgoing message production call from a service
• span.kind: internal - Internal operation within a service

Root spans: A request root span (request.is_root_span == true) represents an incoming call to a service. Use this to analyze end-to-end request performance.

Key Trace Attributes

Essential attributes for trace analysis:

Attribute	Description
trace.id	Unique trace identifier
span.id	Unique span identifier
span.parent_id	Parent span ID (null for root spans)
request.is_root_span	Boolean, true for request entry points
request.is_failed	Boolean, true if request failed
duration	Span duration in nanoseconds
span.timing.cpu	Overall CPU time of the span (stable)
span.timing.cpu_self	CPU time excluding child spans (stable)
dt.smartscape.service	Service Smartscape node ID
dt.service.name	Dynatrace service name derived from service detection rules. It is equal to the Smartscape service node name.
endpoint.name	Endpoint/route name

Service Context

Spans reference services via Smartscape node IDs and the detected service name dt.service.name which is also present on every span.

fetch spans
| summarize spans=count(), by: { dt.smartscape.service, dt.service.name }

Node functions:

• getNodeName(dt.smartscape.service) - Adds dt.smartscape.service.name field with the human-readable service name
• getNodeField(dt.smartscape.service, "attribute_name") - Access specific node attributes

📖 Learn more: See Entity Lookups for advanced entity selectors, infrastructure correlation, and hardware analysis.

Sampling and Extrapolation

One span can represent multiple real operations due to:

• Aggregation: Multiple operations in one span (aggregation.count)
• ATM (Adaptive Traffic Management): Head-based sampling by agent
• ALR (Adaptive Load Reduction): Server-side sampling
• Read Sampling: Query-time sampling via samplingRatio parameter

When to extrapolate: Always extrapolate when counting actual operations (not just spans). Use the multiplicity factor:

fetch spans
| fieldsAdd sampling.probability = (power(2, 56) - coalesce(sampling.threshold, 0)) * power(2, -56)
| fieldsAdd sampling.multiplicity = 1 / sampling.probability
| fieldsAdd multiplicity = coalesce(sampling.multiplicity, 1)
                         * coalesce(aggregation.count, 1)
                         * dt.system.sampling_ratio
| summarize operation_count = sum(multiplicity)

📖 Learn more: See Sampling and Extrapolation for detailed formulas and examples.

Common Query Patterns

Basic Span Access

Fetch spans and explore by type:

fetch spans | limit 1

Explore spans by function and type:

fetch spans
| summarize count(), by: { span.kind, code.namespace, code.function }

Request Root Filtering

List request root spans (incoming service calls):

fetch spans
| filter request.is_root_span == true
| fields trace.id, span.id, start_time, response_time = duration, endpoint.name
| limit 100

Service Performance Summary

Analyze service performance with error rates:

fetch spans
| filter request.is_root_span == true
| summarize
    total_requests = count(),
    failed_requests = countIf(request.is_failed == true),
    avg_duration = avg(duration),
    p95_duration = percentile(duration, 95),
    by: {dt.service.name}
| fieldsAdd error_rate = (failed_requests * 100.0) / total_requests
| sort error_rate desc

Trace ID Lookup

Find all spans in a specific trace:

fetch spans
| filter trace.id == toUid("abc123def456")
| fields span.name, duration, dt.service.name

Performance Analysis

Response Time Percentiles

Calculate percentiles by endpoint:

fetch spans
| filter request.is_root_span == true
| summarize {
    requests=count(),
    avg_duration=avg(duration),
    p95=percentile(duration, 95),
    p99=percentile(duration, 99)
  }, by: { endpoint.name }
| sort p99 desc

💡 Best practice: Use percentiles (p95, p99) over averages for performance insights.

Slow Trace Detection

Find requests exceeding a threshold:

fetch spans, from:now() - 2h
| filter request.is_root_span == true
| filter duration > 5s
| fields trace.id, span.name, dt.service.name, duration
| sort duration desc
| limit 50

Duration Buckets with Exemplars

fetch spans, from:now() - 24h
| filter http.route == "/api/v1/storage/findByISBN"
| summarize {
    spans=count(),
    trace=takeAny(record(start_time, trace.id))
  }, by: { bin(duration, 10ms) }
| fields `bin(duration, 10ms)`, spans, trace.id=trace[trace.id], start_time=trace[start_time]

Performance Timeseries

Extract response time as timeseries:

fetch spans, from:now() - 24h
| filter request.is_root_span == true
| makeTimeseries {
    requests=count(),
    avg_duration=avg(duration),
    p95=percentile(duration, 95),
    p99=percentile(duration, 99)
  }, by: { endpoint.name }

📖 Learn more: See Performance Analysis for advanced patterns and timeseries techniques.

Failure Investigation

Failed Request Summary

Summarize failures by service:

fetch spans
| filter request.is_root_span == true
| summarize
    total = count(),
    failed = countIf(request.is_failed == true),
  by: { dt.service.name }
| fieldsAdd failure_rate = (failed * 100.0) / total
| sort failure_rate desc

Failure Reason Analysis

Breakdown by failure detection reason:

fetch spans
| filter request.is_failed == true and isNotNull(dt.failure_detection.results)
| expand dt.failure_detection.results
| summarize count(), by: { dt.failure_detection.results[reason] }

Failure reasons:

• http_code - HTTP response code triggered failure
• grpc_code - gRPC status code triggered failure
• exception - Exception caused failure
• span_status - Span status indicated failure
• custom_rule - Custom failure detection rule matched

HTTP Code Failures

Find failures by HTTP status code:

fetch spans
| filter request.is_failed == true
| filter iAny(dt.failure_detection.results[][reason] == "http_code")
| summarize count(), by: { http.response.status_code, endpoint.name }
| sort `count()` desc

Recent Failed Requests

List recent failures with details:

fetch spans
| filter request.is_root_span == true and request.is_failed == true
| fields
    start_time,
    trace.id,
    endpoint.name,
    http.response.status_code,
    duration
| sort start_time desc
| limit 100

📖 Learn more: See Failure Detection for exception analysis and custom rule investigation.

Service Dependencies

Service-to-Service Analysis

Analyze service communication patterns:

fetch spans, from:now() - 1h
| filter isNotNull(server.address)
| fieldsAdd
    remote_side = server.address
| summarize
    call_count = count(),
    avg_duration = avg(duration),
    by: {dt.service.name, remote_side}
| sort call_count desc

Outgoing HTTP Calls

Identify external API dependencies:

fetch spans
| filter span.kind == "client" and isNotNull(http.request.method)
| summarize
    calls = count(),
    avg_latency = avg(duration),
    p99_latency = percentile(duration, 99),
  by: { dt.service.name, server.address, server.port }
| sort calls desc

Trace Aggregation

Complete Trace Analysis

Aggregate all spans in a trace to understand full request flow:

fetch spans, from:now() - 30m
| summarize {
    spans = count(),
    client_spans = countIf(span.kind == "client"),

    // Endpoints involved in the trace
    endpoints = toString(arrayRemoveNulls(collectDistinct(endpoint.name))),

    // Extract the first request root in the trace
    trace_root = takeMin(record(
        root_detection_helper = coalesce(
            if(request.is_root_span, 1),
            if(isNull(span.parent_id), 2),
            3),
        start_time, endpoint.name, duration
      ))
}, by: { trace.id }

| fieldsFlatten trace_root
| fieldsRemove trace_root.root_detection_helper, trace_root

| fields
    start_time = trace_root.start_time,
    endpoint = trace_root.endpoint.name,
    response_time = trace_root.duration,
    spans,
    client_spans,
    endpoints,
    trace.id
| sort start_time
| limit 100

Root detection strategy: Use takeMin(record(...)) with a detection helper to reliably find the root request:

1. Priority 1: Spans with request.is_root_span == true
2. Priority 2: Spans without parent (root spans)
3. Priority 3: All other spans

Multi-Service Traces

Find traces spanning multiple services:

fetch spans, from:now() - 1h
| summarize {
    services = collectDistinct(dt.service.name),
    trace_root = takeMin(record(root_detection_helper = coalesce(if(request.is_root_span, 1), 2), endpoint.name))
}, by: { trace.id }
| fieldsAdd service_count = arraySize(services)
| filter service_count > 1
| fields endpoint = trace_root[endpoint.name], service_count, services = toString(services), trace.id
| sort service_count desc
| limit 50

Request-Level Analysis

Request Attributes

Access custom request attributes captured by OneAgent on request root spans:

fetch spans
| filter request.is_root_span == true
| filter isNotNull(request_attribute.PaidAmount)
| makeTimeseries sum(request_attribute.PaidAmount)

Field patterns: request_attribute.<name>, captured_attribute.<name> (always arrays)

→ Request Attributes — full patterns for request attributes, captured attributes, and request ID aggregation

Span Types

Span Type	Detection	Key Fields	Reference
HTTP server (incoming)	span.kind == "server" and isNotNull(http.request.method)	http.route, http.request.method, http.response.status_code	http-spans.md
HTTP client (outgoing)	span.kind == "client" and isNotNull(http.request.method)	server.address, server.port	http-spans.md
Database	span.kind == "client" and isNotNull(db.system)	db.system, db.namespace, db.statement	database-spans.md
Messaging	isNotNull(messaging.system)	messaging.system, messaging.destination.name, messaging.operation.type	messaging-spans.md
RPC / gRPC	isNotNull(rpc.system)	rpc.system, rpc.service, rpc.method, rpc.grpc.status_code	rpc-spans.md
Serverless / FaaS	isNotNull(faas.name) and span.kind == "server"	faas.name, faas.trigger.type, cloud.provider	serverless-spans.md

⚠️ Database spans: Can be aggregated (one span = multiple calls). Always use aggregation.count extrapolation for accurate operation counts.

📖 Detailed patterns per span type: See the reference files above.

Advanced Topics

Exception Analysis

Exceptions are stored as span.events within spans:

fetch spans
| filter iAny(span.events[][span_event.name] == "exception")
| expand span.events
| fieldsFlatten span.events, fields: { exception.type }
| summarize {
    count(),
    trace=takeAny(record(start_time, trace.id))
  }, by: { exception.type }
| fields exception.type, `count()`, trace.id=trace[trace.id], start_time=trace[start_time]

💡 Tip: Use iAny() to check conditions within span event arrays.

→ Logs Correlation — joining logs and traces, filtering traces by log content → Network Analysis — client IPs, DNS resolution, subnet analysis

Best Practices

Area	Rule
Filtering	Apply request.is_root_span == true and endpoint filters first
Sampling	Use samplingRatio (e.g., 100 = read 1%) for performance
Percentiles	Use p95/p99 over averages for performance analysis
Root spans	Use request.is_root_span == true for end-to-end analysis
Trace grouping	Group by trace.id for complete trace metrics
Request grouping	Group by request.id for OneAgent-only request metrics
Extrapolation	Always apply multiplicity for accurate operation counts
Exemplars	Use takeAny(record(start_time, trace.id)) to enable UI drilldown

---

Troubleshooting

Problem	Cause	Solution
Duration values seem wrong (too large)	duration is in nanoseconds, not milliseconds	Divide by 1000000 or compare with 5s (DQL duration literal)
Span counts don’t match expected request volume	Sampling or aggregation not accounted for	Use multiplicity extrapolation — see Sampling and Extrapolation reference
getNodeName(dt.smartscape.service) returns null	Service not yet resolved or OneAgent not monitoring	Verify OneAgent monitors the service; entity resolution may have a short delay
request.is_root_span filter returns nothing	Querying OpenTelemetry-only traces without OneAgent	Use isNull(span.parent_id) as fallback for root span detection
trace.id filter returns no results	Trace ID not converted to UID format	Use filter trace.id == toUid("abc123...") for string-based trace IDs
Database span counts are too low	Database spans are aggregated (one span = N calls)	Always use aggregation.count extrapolation for database operation counts

Related Skills

• dt-dql-essentials — Core DQL syntax for querying trace data
• dt-app-dashboards — Embed trace queries in dashboards
• dt-migration — Smartscape entity model and relationship navigation

---

References

Detailed documentation for specific topics:

• Performance Analysis - Advanced timeseries, duration buckets, endpoint ranking
• Failure Detection - Failure reasons, exception investigation, custom rules
• Sampling and Extrapolation - Multiplicity calculation, database extrapolation
• Request Attributes - Request attributes, captured attributes, request ID aggregation
• Entity Lookups - Advanced node lookups, infrastructure correlation, hardware analysis
• HTTP Span Analysis - Status codes, payload analysis, client IPs
• Database Span Analysis - Extrapolated counts, slow queries, statement analysis
• Messaging Span Analysis - Kafka, RabbitMQ, SQS throughput and latency
• RPC Span Analysis - gRPC, SOAP, service dependencies
• Serverless Span Analysis - Lambda, Azure Functions, cold start analysis
• Logs Correlation - Joining logs and traces, correlation patterns
• Network Analysis - IP addresses, DNS resolution, communication mapping