OTLP ingestion overview

This document introduces the use of the Telemetry (OTLP) API, telemetry.googleapis.com, which implements the OpenTelemetry Protocol. The Telemetry API lets you ingest OTLP-formatted log, metric, and trace data into Google Cloud Observability:

• OTLP log records are converted into log entries and then routed and stored. For information about the conversion process, see the OTLP logs ingestion section of this document.
• Metric data is ingested into Cloud Monitoring. For information about metric and label names and the ingestion restrictions, see the OTLP metrics ingestion section of this document.
• Trace data is stored in a format that is generally consistent with OTLP. For more information, see OTLP trace ingestion.

You can send telemetry data to the Telemetry API from applications that use SDKs or by exporting from an OpenTelemetry Collector.

If you are using Google Kubernetes Engine, then you can use Managed OpenTelemetry for GKE instead of manually deploying and configuring an OpenTelemetry Collector that uses the Telemetry API.

Protocol support

The OTLP endpoint supports all of the OTLP transport and serialization protocols, including http/protobuf, http/json, and grpc. When exporting directly from applications using SDKs, we recommend using the gRPC OTLP exporter rather than HTTP exporters because most SDK exporters lack support for dynamic token refreshing.

Authentication

You must configure your exporters with the credentials necessary to send data to your Google Cloud project. For example, when you use collectors, typically you use the googleclientauth extension to authenticate with Google credentials.

For an example of authentication when using direct export of trace data, see Configure authentication. This example illustrates how to configure the exporter with your Google Cloud Application Default Credentials (ADC) and add a language-specific Google Auth Library to your application.

To send telemetry data to your Google Cloud project by using the Telemetry API, you must also do the following:

• Configure a quota project. To learn more, see Set the quota project.

• Grant the following Identity and Access Management (IAM) roles to the user or to the service account that the application uses:

  • Service Usage Consumer role (roles/serviceusage.serviceUsageConsumer) on the quota project.
  • Cloud Telemetry Writer role (roles/telemetry.writer) on the project. This role lets your application write log, metric, and trace data.

OTLP ingestion

This section describes how your log, metric, and trace data is converted from OTLP into Google Cloud Observability data structures.

Log data ingestion

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

When you use the Telemetry API to ingest OTLP-formatted logs, your log data is converted into Cloud Logging log entries. An incoming OTLP-formatted log request in JSON has the following general structure:

"resourceLogs": [
    {
      "resource": {
        "attributes": [...]
      },
      "scopeLogs": [
        {
          "scope": { ...}
          "logRecords": [...]
        }
      ]
    }
]

Each item in each logRecords array becomes a single Cloud Logging log entry. The resource attributes determine the monitored resource in the resulting LogEntry. For more information about which attributes are required for ingestion of OTLP-formatted logs, see OTLP attributes to resource-type mapping.

To support ingestion of OTLP-formatted logs, the Cloud Logging LogEntry structure contains an additional field, otel. Because the OTLP and Cloud Logging data models differ in structure, the otel field preserves a copy of the resource, scope, and entity metadata from the incoming OTLP request.

For example, if you send an OTLP resourceLogs payload like the following to the Telemetry API, then each resulting log entry contains a resource field (for the monitored resource) and an otel field, as shown on the other tabs:

resourceLogs

{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "gcp.project_id",
            "value": { "stringValue": "PROJECT_ID" }
          },
          {
            "key": "gcp.resource_type",
            "value": { "stringValue": "global" }
          }
        ]
      },
      "scopeLogs": [
        {
          "scope": {
            "name": "my.library",
            "version": "1.0.0",
            "attributes": [
              {
                "key": "my.scope.attribute",
                "value": { "stringValue": "some scope attribute" }
              }
            ]
          },
          "logRecords": [ ... ]
         }
       ]
     }
   ]
}

resource

  {
    ...
    "resource": {
      "labels": {
        "project_id": "PROJECT_ID"
      },
      "type": "global"
    },
    ...
}

otel

  {
    ...
    "otel": {
      "resource": {
        "attributes": {
          "gcp.project_id": "PROJECT_ID",
          "gcp.resource_type": "global"
        }
      },
      "scope": {
        "attributes": {
          "my.scope.attribute": "some scope attribute"
        },
        "name": "my.library",
        "version": "1.0.0"
      }
    },
   ...
  }

Because Cloud Logging log entries are self-contained and don't link to external resource schemas, all OTLP resource, scope, and entity metadata is copied into each log entry.

Metric data ingestion

OTLP for Prometheus metrics only works when using the OpenTelemetry Collector version 0.140.0 or newer.

When metrics are ingested into Cloud Monitoring by using an OpenTelemetry Collector and the otlphttp exporter or sent directly by using an OpenTelemetry SDK, the OTLP metrics are mapped to Cloud Monitoring metric structures. To get information about those mappings, see the following:

• Mapping between OTLP resources and Cloud Monitoring monitored resources.
• Mapping between OTLP metrics and Cloud Monitoring metrics.

Google Cloud Observability converts metrics into the Prometheus time-series format. Metric names must have either no domain or the domain prometheus.googleapis.com. After conversion, the metric name includes the prometheus.googleapis.com prefix and an additional suffix, based on the OTLP point kind. The resulting Cloud Monitoring metric has the following structure:

prometheus.googleapis.com/{metric_name}/{suffix}

Additionally, for each unique OpenTelemetry resource, the conversion adds a target_info metric that contains all resource attributes except service.name, service.instance.id, and service.namespace.

Note: Value-type collisions can occur if you previously sent INT64 target_info metrics. If you encounter collisions, the best long-term solution for this is for you to delete the metric descriptor for the INT64 target_info metrics. To delete this descriptor from your metric scope, use this Golang script.

Because metric names and label keys in Cloud Monitoring don't support full UTF-8, metric data can be rejected:

• Metric names that don't conform to the regular expression [a-zA-Z][a-zA-Z0-9_:./-]* are rejected. The only special characters allowed in metric names are in the set _:./-.
• Data points containing attributes (that is, label keys) that don't conform to the regular expression [a-zA-Z_][a-zA-Z0-9_.]* are rejected. The only special characters allowed in label keys are in the set _.. All special characters are allowed in label values.

To prevent rejection of your metrics for these reasons, use the replace_pattern function to transform your metric names and attributes.

Trace data ingestion

Regardless of whether you use the Telemetry API or the Cloud Trace API, incoming trace data is stored in a format consistent with OTLP. However, we recommend using the Telemetry API because it provides higher ingestion quotas than the Cloud Trace API.

The following is an example of trace data that might be sent from an application to your Google Cloud project:

{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [...]
      },
      "scopeSpans": [
        {
          "scope": { ...},
          "spans": [...]
        }
      ]
    }
  ]
}

Each item in each scopeSpans.spans array becomes a single stored span:

• Each span's resource field contains a copy of the resourceSpans.resource.attributes data.
• Each span's instrumentation_scope field contains a copy of the scopeSpans.scope data.
• Each span corresponds to one entry in the scopeSpans.spans array. Fields such as traceId, spanId, and kind are mapped into similarly named fields in the trace schema.

For more information, see the following documents:

• Storage schema for trace data.
• v1.traces overview.

Billing

Billing for log, metric, and trace data that is ingested by using the Telemetry API depends on the telemetry signal. For complete information, see the Billing page.

Log data billing

You might see a change in your Cloud Logging storage and billing values when you use the Telemetry API to ingest logs due to a change in the log volume.

The largest changes to storage and billing for your Google Cloud project occur when both of the following are true:

• The resource field contains high-cardinality attributes or a large number of attributes. These resource attributes determine the monitored resource in the resulting LogEntry.
• The scopeLogs field contains a large number of items in the logRecords arrays. The scopeLogs.scope fields are copied into the otel field for every individual log entry.

Because this resource and scope metadata is copied onto every individual log entry, your stored log volume can increase.

To minimize storage volume, we recommend the following:

• Use an OpenTelemetry Collector processor, such as a transform processor, to drop unnecessary resource or scope attributes before exporting the data.
• If you don't need the additional metadata preserved in the otel field, then use the legacy mapping option, gcp.use_legacy_mapping, which prevents the otel field from being populated.

Note: If you plan to use the volume of the resourceLogs object to estimate the volume of the resulting LogEntry objects, then send one logRecord object per resourceLogs object. Otherwise, your estimates won't properly account for fields that are duplicated in each LogEntry object.

Metric data billing

Billing for OTLP metrics is accounted for under the "Prometheus Samples Ingested" SKU, the same one used for metrics from Google Cloud Managed Service for Prometheus.

Trace data billing

The API that you use to send trace data to your project doesn't affect how charges are computed for that data.

Querying your log, metric, and trace data

You can use the explorer pages—Logs Explorer, Metrics Explorer, and Trace Explorer—to query your log, metric, and trace data. You can also use the Observability Analytics page to analyze your log and trace data by using SQL.

The following tips might be helpful when you query your metric data using the Metrics Explorer:

• Important: Querying metric names and label keys with special characters other than the colon (:) and the underscore (_) requires you to wrap them in braces ({}) and quotes ("), according to PromQL's UTF-8 spec. For example, the following are valid queries:

  • {"my.metric.name"}
  • {"my.metric.name", "label.key.KEY"="value"}

• Retaining the le label when querying exponential histograms might return unexpected results. The more typical histogram_quantile(.99, sum by (le) (metric)) queries are expected to work.

• Delta metrics might not query properly in certain circumstances, such as very sparse deltas.

Limits and quotas

The Telemetry API limits apply to all signal types.

The following quotas and limits also apply:

• Log data: The Cloud Logging API quotas and limits apply.

• Metric data: The Cloud Monitoring API quotas and limits apply. For example, metrics can't have more than 200 labels.

  The default quota for metrics ingested by the Telemetry API is 60,000 requests per minute. At a maximum batch size of 200 points per request, this quota is an effective default quota of 200,000 samples per second. You can request a quota increase.

• Trace data: There are no additional quotas or limits that apply.

What's next

• For information about migrating to the otlphttp exporter from another exporter, see Migrate to the OTLP exporter.
• For instructions about deploying and using the OpenTelemetry Collector with the Telemetry API, see Deploy and use the collector.
• For information about writing OTLP-formatted logs to the Telemetry API, see Write OTLP-formatted logs to the Telemetry API.
• For information about sending metrics to the Telemetry API from applications that use SDKs, see Use SDKs to send metrics from applications.
• For information about using an OpenTelemetry Collector and the Telemetry API with OpenTelemetry zero-code instrumentation, see Use OpenTelemetry zero-code instrumentation for Java.