This is an overview of the main concepts around OpenTelemetry.
You can take a look at the Opentelemetry documentation to see the OpenTelemetry Components.
Summary
The OpenTelemetry (OTEL) Collector offers a vendor-agnostic implementation of how to receive, process and export telemetry data.
The Otel Collector can collect data from OpenTelemetry SDKs and other sources (in multiple formats, for example, OTLP, Jaeger, Prometheus, as well as many commercial/proprietary tools), and then export this telemetry to any supported backend, such as Jaeger, Prometheus or Kafka queues. It also supports processing and filtering telemetry data before it gets exported.
Basically, it's a middleware that can run either very close to your workloads (“agent” mode), or on a central location (“collector”, or “service” mode - as a central collector service aggregating across multiple application nodes). For more information on the deployment see Patterns you can apply to deploy the OpenTelemetry collector.
By default the Collector configuration is located at a config.yaml file (see Location Configuration).
It is built following a pipeline principle, with receivers, processors, and exporters, potentially making use of extensions:
- receivers are the components that are responsible for gathering or receiving telemetry data.
- processors are able to look into the telemetry data and take decisions based on them, such as perform data cleanup, apply sampling decisions, include or remove attributes, enrich data based on where it’s running, among others.
- exporters will then send the data out of the process, likely to one or more telemetry backends or another layer of collectors.
So an exporter is how data gets sent to different systems/back-ends. Generally, an exporter translates the internal format into another defined format.
See OpenTelemetry documentation for Exporters in Java and NET.
- extensions are not part of the data pipeline but can help other components like adding extensions for Collector health monitoring.
OpenTelemetry Protocol (OTLP) specifies how OpenTelemetry-related components exchange telemetry data.
This includes instrumented applications, intermediary services such as the Collector, as well as backends to which the data is eventually sent.
OTLP primarily defines message structures, encoding, and endpoints and relies on HTTP or gRPC as the underlying transport protocols.
The following are some enviroment variables that are generally used to configure the OTLP Exporter.
- OTEL_EXPORTER_OTLP_PROTOCOL - The transport protocol (grpc, http/protobuf, http/json).
- OTEL_EXPORTER_OTLP_ENDPOINT - Endpoint (OTLP/HTTP or OTLP/gRPC).
Check OTLP Exporter configuration.
You can define environment variables for setting one or more exporters per signal.
- OTEL_TRACES_EXPORTER
- OTEL_METRICS_EXPORTER
- OTEL_LOGS_EXPORTER
In general you also define:
- OTEL_SERVICE_NAME - Sets the value of the service.name resource attribute
- OTEL_RESOURCE_ATTRIBUTES - Key-value pairs to be used as resource attributes
Check General SDK configuration documentation.
For testing and debugging see Debug Exporter.
Besides, OpenTelemetry requires a backend system to store, process, analyze, and visualize the telemetry data collected from applications. It does not include a specific built-in backend to process and analyze the data. Instead, it allows to integrate with various backend systems.
Backend systems receive the exported telemetry data from the instrumented applications or the OpenTelemetry Collector. These systems are responsible for storing, analyzing, and visualizing the telemetry data. Ex: Prometheus, AWS Xray, Zipkin, Lightstep, etc.
There are also visualization services for viewing data that is connected to Backend services. Ex: Grafana