When OpenTelemetry first came into the picture with the merger of OpenCensus and OpenTracing in 2019, it was pretty much all about classic telemetry data, namely- logs, metrics, and traces.
Since then, OpenTelemetry has become an indispensable tool in the modern observability landscape. With frequent integrations and introduction to new capabilities every year or so, it has poised itself as an invaluable tool for cloud enterprises.
Later on, Otel announced the capability of “Profiling”. This blog is a focused comparison of profiling vs tracing in OpenTelemetry, including how they complement each other and when to use each.
OpenTelemerty: A Quick Overview
Engineered as a comprehensive toolkit for collecting telemetry data, including metrics, logs, and traces—OpenTelemetry simplifies the process of application instrumentation, empowering developers to gain deeper insights into their applications’ performance.
Simply put, OpenTelemetry is an open-source project that provides a unified set of tools, APIs, and SDKs to collect and export telemetry data (metrics, logs, and traces) from cloud-native software applications.
However, that definition is going to be a bit longer in the coming years. Why? Because OTel is evolving, and for much-needed reasons, given the rapid evolution of data and the way it’s being managed. Not to mention the ubiquity of AI in recent times.
Presently, OpenTelemetry stands as the 2nd most active project within the Cloud Native Computing Foundation (CNCF), highlighting its escalating significance in the realm of observability.
What is Profiling?
Profiling in OpenTelemetry, particularly continuous profiling, aims to provide deep insights into resource consumption and code-level performance within an application. It involves:
- Collecting Data:Gathering detailed information about CPU usage, memory allocation, I/O operations, and function call times.
- Identifying Hotspots:Pinpointing specific functions or code segments that are consuming the most resources or causing performance degradation.
- Purpose:Optimizing resource utilization, identifying inefficient code, and understanding the root cause of performance bottlenecks at a granular level.
Profiling helps identify areas for optimization, particularly in terms of CPU usage, memory consumption, and execution time hot spots, by capturing instances of the code at various intervals.
Moreover, sustained profiling not only sheds light on resource utilization at the code level but also enables the storage, querying, and analysis of this data over time.
Precisely, OpenTelemetry profiling captures low-level performance behavior inside services:
Records insights such as CPU usage, memory allocation, function execution time, I/O waits, and thread activity.
Can run continuously (via eBPF or runtime agents) or on-demand to spot code-level inefficiencies.
Best at answering “why is something slow or resource-hungry?”—pinpointing code paths or functions causing issues.
What is Tracing?
- Spans:Individual units of work or operations within a trace, representing a specific action or period.
- Trace Context:Information that links spans together, forming a complete end-to-end view of a request’s journey across multiple services.
- Purpose:Identifying latency issues, bottlenecks at the service level, and understanding the overall execution path of a request.
This allows developers and operations teams to visualize the sequence and interaction of operations, helping them diagnose issues, optimize performance, and understand system behavior in complex, distributed environments.
In short, OpenTelemetry tracing captures the execution path of requests as they move through distributed systems:
Tracks spans across services, showing each step—from client request to database interaction, external API call, etc.
Helps identify latency, bottlenecks, errors, and request flow issues across microservices.
Best at answering “what happened and where in the system?”, enabling root cause debugging in complex architectures.
Understanding the Difference: Tracing and Profiling
While tracing is at the core of OTel, think of profiling as an augmentation to the capabilities. Let’s have a look at the differences thoroughly:
Tracing in OpenTelemetry provides a means to track the journey of a request through various services and processes. It helps developers understand the flow of requests, identify bottlenecks, and pinpoint failures in a distributed system.
Additionally, Traces are composed of spans, where each span represents a single operation or unit of work. This makes it easier to visualize the path of a request and observe how different parts of the system interact.
Profiling, on the other hand, is slightly different. While it’s a newer addition to OpenTelemetry, profiling offers a way to measure where a program spends its time or uses other resources like CPU or memory.
Profilers run in the background and can provide a granular view of resource usage over time, making them invaluable for optimizing performance and understanding system behavior under different load conditions.
Differences at a Glance
| Signal | Purpose | Granularity | Primary Use Case |
|---|---|---|---|
| Tracing | Tracks execution paths across services/components to pinpoint where latency or failures occur | Coarse: spans and trace durations | Debugging inter-service latency, request flow, and bottlenecks |
| Profiling | Monitors runtime behavior (CPU, memory, function execution) to identify inefficient code paths | Fine: stack traces, function-level resource usage | Optimizing hotspots, resource usage, root-cause code inefficiencies |
| Combined Use | Use tracing to find the problematic span, then profile within that span to find the exact line or function causing delays | ||
The Complementary Nature of Profiling and Tracing
In OpenTelemtery, tracing and profiling serve complementary purposes in application monitoring and performance optimization:
– Tracing: provides a high-level overview of request paths and interactions within your distributed system, making it essential for identifying and diagnosing systemic issues such as latency and errors.
– Profiling: dives deeper into individual processes, offering detailed insights into resource usage and potential inefficiencies at the code level.
By combining these two techniques, you get a well-rounded view of application performance.
To put into perspective:
- Tracing (Macro-level): Shows you the big picture – how requests flow and where delays might be happening.
- Profiling (Micro-level): Shows you the granular details – why a specific code section is slow and what resources it’s consuming.
This comprehensive approach is especially valuable for developers and performance engineers. They can use it to:
- Understand exactly how the application functions.
- Identify root causes of performance problems.
- Optimize code for better efficiency.
Real World Scenario
A user request takes 500 ms.
Tracing shows it hits Service A → Service B → Database, with Service B causing most delay.
Profiling within Service B reveals a particular function hogs CPU and blocks I/O during that span.
You optimize that function, reducing processing time—both trace latency and resource usage improve.
What’s Next for OpenTelemetry?
Looking ahead to 2025, OpenTelemetry is poised to expand its capabilities significantly. The focus will be on stabilizing the current features and introducing advanced ones like profiling and client-side Real User Monitoring (RUM). These additions are expected to provide developers and organizations with even more tools to monitor and optimize their applications effectively.
OpenTelemetry has significantly transformed the way telemetry data is aggregated by standardizing data formats across different technologies. This uniform format simplifies the processes of working with, combining, and analyzing data.
The introduction of profiling as a signal type is particularly exciting because it would further enhance this standardization, enabling a unified approach to understanding and optimizing code performance across diverse environments and languages. This development promises to make telemetry data even more accessible and actionable for users.
OpenTelemetry now officially includes profiling as a signal in OTLP (since v1.3.0), although it remains experimental and evolving. Collector support and semantic conventions are available but not yet considered stable.
Many observability platforms are already leveraging profiling + tracing correlation (e.g., Apica, Grafana, Elastic), enabling actionable insights into app performance.
When to Use Profiling or Tracing
Use tracing when you want to map the path of a request across services or microservices.
Use profiling when you need deep diagnostics on the performance of code itself: memory leaks, CPU spikes, slow loops, etc.
Use both together when debugging slow request execution: trace first to spot a problematic span, then inspect profiling data in that span to find the guilty code path.
TL;DR
Profiling: Enhances application insights by monitoring runtime behavior and identifying optimization needs.
Tracing: Tracks and documents request execution paths across services, aiding in performance diagnostics.
Differences and Synergies: Tracing offers system-level views; profiling provides deep dives into resource and code performance.
Future Directions: Plans to expand features, including profiling and Real User Monitoring (RUM) for comprehensive insights.
