Datadog pipeline observability error with source as gcp pub/sub and destination as Rsyslog

 Datadog pipeline observability error with source as gcp pub/sub and destination as Rsyslog

pods panic error before it dies
An error occurred that Vector couldn't handle: receiver disconnected.
The exact panic error
"Tried to call 'Value::as_timestamp_unwrap' on a non-timestamp" exposes a known data-type validation bug inside the Datadog Observability Pipeline Worker's (Vector) engine when format translation is happening

This panic means that the pipeline code is processing a log event and strictly expects its timestamp field to be native Rust/Vector Timestamp metadata object, but it is encountering a different type instead—most likely a raw string or an integer. [1]

Because it calls _unwrap() under the hood without a safe error fallback pattern, the engine panics and causes the GKE pod to crash instantly.

Why did adding DD_OP_DESTINATION_SYSLOG_ENDPOINT_URL trigger this?

When you route a pipeline to a Syslog destination, the Datadog engine automatically injects a hidden Syslog Formatter into the data path. This formatter intercepts incoming events (from your GCP Pub/Sub source) and parses them to build a RFC-5424 or RFC-3164 compliant syslog header, which strictly requires an formatted timestamp. [1]

If your incoming logs from GCP Pub/Sub have overridden, modified, or omitted the root-level @timestampmetadata field (storing it instead as a plain text JSON string), the Syslog translation layer tries to parse it, fails, and panics. [1]

---

Datadog console Fix

Fix 1: Add a VRL Transform to coerce the Timestamp Type

If you are managing your pipeline components through the Datadog Remote UI (or a local configuration file), you must explicitly force the string field back into a recognized timestamp data type before it gets sent to the rsyslog destination.

Add a Remap / Vector Remap Language (VRL) component right after your GCP Pub/Sub source:

vrl

# Force parse the text timestamp into a proper native timestamp type
if exists(.timestamp) {
  .timestamp = parse_timestamp!(string!(.timestamp), format: "%Y-%m-%dT%H:%M:%S%.fZ")
} else if exists(.@timestamp) {
  .@timestamp = parse_timestamp!(string!(.@timestamp), format: "%Y-%m-%dT%H:%M:%S%.fZ")
}

Use code with caution.

(Adjust the string format placeholder above to match whatever time string format your GCP logs currently output, e.g., RFC3339).

Fix 2: Drop Custom Mappings on the Datadog Syslog Destination Component

If you configured custom structural attributes under your Syslog Destination Settings via the Datadog UI:

1. Open the Datadog Observability Pipelines UI dashboard.
2. Select your pipeline configuration and click edit on your Syslog Destination.
3. Check the mapping properties. If you customized the Timestamp Key mapping option (e.g., trying to map it to a sub-nested payload key like ://), remove it.
4. Fall back to letting Datadog use the default root event timestamp metadata.

Fix 3: Change the Syslog Framing Format

Some older rsyslog endpoints struggle with structured RFC-5424 metadata payloads. If the crash persists, change the payload format parameter inside your destination setup from rfc5424 to rfc3164 or use raw text string mode. This changes how the internal encoder evaluates time tokens, bypassing the internal strict unwrap requirements. [1]

Helm Chart fix

To fix this bug when deploying via the Datadog Observability Pipelines Worker (OPW) Helm Chart, you must manually define the pipeline components inside your values.yaml file.

You need to insert a Remap (VRL) transformation block directly between your GCP Pub/Sub source and your Rsyslog destination. This converts the plain text string from GCP into a native timestamp object before the Syslog encoder touches it, completely bypassing the internal Rust panic code.

Here is the exact structure required in your Helm values.yaml configuration:

yaml

# values.yaml
pipeline:
  # 1. Your existing GCP Pub/Sub Source
  sources:
    gcp_pubsub_source:
      type: "gcp_pubsub"
      project: "your-gcp-project-id"
      subscription: "your-pubsub-subscription-name"

  # 2. THE FIX: Insert this Remap step to cast the type safely
  transforms:
    fix_timestamp_type:
      type: "remap"
      inputs:
        - "gcp_pubsub_source"
      source: |
        # Check if timestamp exists and convert it from String to native Timestamp object
        if exists(.timestamp) {
          parsed, err = parse_timestamp(string!(.timestamp), format: "%Y-%m-%dT%H:%M:%S%.fZ")
          if err != null {
            # Fallback to the current system time if the GCP format doesn't match perfectly
            .timestamp = to_timestamp(now())
          } else {
            .timestamp = parsed
          }
        } else {
          # Inject a fallback timestamp if the field is entirely missing
          .timestamp = to_timestamp(now())
        }

  # 3. Your Rsyslog Destination (Using the transform as its input)
  destinations:
    rsyslog_destination:
      type: "syslog"
      inputs:
        - "fix_timestamp_type" # <-- Must pull from the transform, NOT the source directly
      endpoint: "rsyslog-service.local:514" # Ensure NO 'tcp://' or 'udp://' prefix
      mode: "tcp" # Mode parameter inside the values block works flawless