Agent Skill · dynatrace

dt-obs-services

Service performance monitoring with RED metrics (Rate, Errors, Duration) and runtime-specific telemetry for Java, .NET, Node.js, Python, PHP, and Go. Use when analyzing service health, SLA compliance, or runtime issues. Trigger: "service response time", "error rate", "throughput", "SLA compliance", "service mesh overhead", "JVM GC", "Java heap", "Node.js event loop", ".NET CLR", "Python threads", "PHP OPcache", "Go goroutines", "service performance", "p95 latency", "request failures", "database response time by name". Do NOT use for explaining existing queries, product documentation questions, infrastructure metrics (use dt-obs-hosts), log analysis (use dt-obs-logs), or distributed tracing workflows (use dt-obs-tracing).

View SKILL.md on GitHub → Source repository Provider profile
Provider: dynatrace Path in repo: skills/dt-obs-services/SKILL.md

Skill body

Application Services Skill

Monitor application service performance, health, and runtime-specific metrics using DQL.

Core Capabilities

1. Service Performance (RED Metrics)

Monitor service Rate, Errors, Duration using metrics-based timeseries queries.

Key Metrics:

Common Use Cases:

Quick Example:

timeseries {
  p95 = percentile(dt.service.request.response_time, 95),
  total_requests = sum(dt.service.request.count),
  failures = sum(dt.service.request.failure_count)
}, by: {dt.service.name}
| fieldsAdd p95_ms = p95[] / 1000, error_rate_pct = (failures[] * 100.0) / total_requests[]

→ For detailed queries: See references/service-metrics.md

2. Advanced Service Analysis

Span-based queries for complex scenarios requiring flexible filtering and custom aggregations.

Use Cases:

Quick Example:

fetch spans, from: now() - 1h | filter request.is_root_span == true
| fieldsAdd meets_sla = if(request.is_failed == false AND duration < 3s, 1, else: 0)
| summarize total = count(), sla_compliant = sum(meets_sla), by: {dt.service.name}
| fieldsAdd sla_compliance_pct = (sla_compliant * 100.0) / total

→ For detailed queries: See references/service-metrics.md

3. Service Messaging Metrics

Monitor message-based service communication (queues, topics).

Key Metrics:

Use Cases:

Quick Example:

timeseries {
  published = sum(dt.service.messaging.publish.count),
  received = sum(dt.service.messaging.receive.count),
  processed = sum(dt.service.messaging.process.count),
  failed = sum(dt.service.messaging.process.failure_count)
}, by: {dt.service.name}

→ For detailed queries: See references/service-metrics.md

4. Service Mesh Monitoring

Monitor service mesh ingress performance and overhead.

Key Metrics:

Use Cases:

Quick Example:

timeseries {
  direct_p95 = percentile(dt.service.request.response_time, 95),
  mesh_p95 = percentile(dt.service.request.service_mesh.response_time, 95)
}, by: {dt.service.name}
| fieldsAdd mesh_overhead_ms = (mesh_p95[] - direct_p95[]) / 1000

→ For detailed queries: See references/service-metrics.md

5. Runtime-Specific Monitoring

Technology-specific runtime performance and resource usage metrics.

Java/JVM - references/java.md

Node.js - references/nodejs.md

.NET CLR - references/dotnet.md

Python - references/python.md

PHP - references/php.md

Go - references/go.md

When to Use This Skill

✅ Use for:

❌ Don’t use for:

Agent Instructions

Act First, Refine Later

When a user asks for analysis — threshold checks, anomaly detection, performance comparisons — proceed immediately with sensible defaults. Do not ask the user for parameter values you can reasonably assume.

Why this matters: analysis tools (e.g., static-threshold-analyzer) require specific inputs like threshold values and service scope. The user expects results, not a parameter interview. Pick reasonable defaults, state them clearly in the response, and let the user refine.

Default values when not specified:

Parameter Default Rationale
Response time threshold 1000 ms (= 1,000,000 µs in the metric’s base unit) Common SLA boundary
Service scope All services Show the most relevant violations
Timeframe From the request, or last 30 min for threshold checks, 2h for general analysis Matches typical operational windows

Example: threshold violation request

  1. Use create-dql to build a timeseries query for avg(dt.service.request.response_time) grouped by dt.smartscape.service
  2. Pass the query to static-threshold-analyzer with threshold = 1000000 (µs), alertCondition = ABOVE
  3. Resolve entity IDs to names using get-entity-name
  4. Present violations with service names, timestamps, values, and duration

Reading user phrasing: Phrases like “the fixed threshold”, “a threshold”, or “the limit” name the type of analysis — static threshold check — not a specific number the user expects you to already know. “Fixed” distinguishes a static cutoff from a dynamic or seasonal baseline. When you see these phrases, apply the 1000 ms default from the table above and present results — the user can then refine if the default doesn’t match their intent.

Scope Boundary

This skill covers service performance metrics and runtime monitoring only. If the user asks a product documentation or configuration question (e.g., “How do I add custom sensors?”, “How do I configure service detection?”), use ask-dynatrace-docs instead — this skill does not contain configuration how-tos.

Understanding User Intent

Map user questions to capabilities:

User Request Use Capability Key Files
“service performance”, “response time”, “error rate” Service Performance (RED) service-metrics.md
“SLA tracking”, “health scoring” Advanced Service Analysis service-metrics.md
“service mesh”, “Istio”, “Linkerd”, “mesh overhead” Service Mesh Monitoring service-metrics.md
“messaging”, “queue”, “topic”, “publish”, “consumer” Service Messaging Metrics service-metrics.md
“JVM GC”, “Java memory”, “heap” Runtime-Specific (Java) java.md
“Node.js event loop”, “V8 heap” Runtime-Specific (Node.js) nodejs.md
“.NET CLR”, “GC generation” Runtime-Specific (.NET) dotnet.md
“Python GC”, “thread count” Runtime-Specific (Python) python.md
“OPcache”, “PHP GC” Runtime-Specific (PHP) php.md
“goroutines”, “Go GC”, “scheduler” Runtime-Specific (Go) go.md

Query Construction Patterns

1. Metrics-based (timeseries)

2. Span-based (fetch spans)

3. Comparison queries

Response Construction Guidelines

Always include:

  1. Metric name(s) - Clear metric identifiers
  2. Aggregation - How data is aggregated (avg, sum, percentile)
  3. Grouping - Dimensions used (dt.service.name, k8s.workload.name, etc.)
  4. Unit conversion - Convert microseconds to milliseconds where appropriate
  5. Filtering - Relevant thresholds or conditions

When referencing runtime-specific content:

Common Workflows

Workflow: Service Health Check

1. Check response time (RED metrics)
2. Check error rate (RED metrics)
3. Check traffic patterns (RED metrics)
4. If runtime-specific issues suspected → Load runtime-specific reference

Workflow: SLA Monitoring

1. Define SLA criteria (e.g., < 3s response time AND < 1% error rate)
2. Use span-based query for custom SLA logic
3. Calculate compliance percentage
4. Filter non-compliant services

Workflow: Service Mesh Analysis

1. Check mesh response time
2. Compare mesh vs direct performance
3. Calculate mesh overhead
4. Analyze mesh failure rates

Workflow: Runtime Troubleshooting

  1. Identify technology stack → Load runtime-specific reference
  2. Check memory/GC metrics → threads/goroutines → runtime features

Troubleshooting

Problem Cause Solution
Response time values look too large Metric is in microseconds Divide by 1000 to convert to milliseconds
No data for service mesh metrics Service mesh not configured Verify mesh sidecar injection is enabled
Runtime metrics missing Wrong technology or no OneAgent Confirm the runtime is supported and OneAgent is active
dt.smartscape.service returns SmartscapeId, not name Need entity name resolution Use getNodeName(dt.smartscape.service)
Error rate always zero Using wrong failure metric Use dt.service.request.failure_count, not custom fields

References

Core Service Monitoring:

Runtime-Specific Monitoring:

Skill frontmatter

license: Apache-2.0