summaryrefslogtreecommitdiff
path: root/docs/core
diff options
context:
space:
mode:
authorJerop Kipruto <[email protected]>2025-06-13 20:28:18 -0700
committerGitHub <[email protected]>2025-06-13 23:28:18 -0400
commit1452bb4ca4ffe3b5c13aab81baaf510d4c45f06f (patch)
treeb61768b4dcfd0ec783798be22877a721487c1e24 /docs/core
parentdefb0fac2cf0bfd86f9696336f12b05493059a27 (diff)
Add GCP telemetry script (#1033)
Adds a script - `scripts/telemetry_gcp.js` - to simplify setting up a local OpenTelemetry collector that forwards data to Google Cloud. This is a follow up to the script for local telemetry `scripts/local_telemetry.js` added in #1015. This script automates downloading necessary binaries, configuring the collector, and updating workspace settings. Also includes `scripts/telemetry_utils.js` with shared helper functions for telemetry scripts. Will refactor `scripts/local_t elemetry.js` in next steps to use this shared functionality. Updates `docs/core/telemetry.md` to include: - A new "Quick Start" section - Detailed instructions for the new GCP automated script - Reorganization of existing sections for clarity #750 --- ``` โœจ Starting Local Telemetry Exporter for Google Cloud โœจ โš™๏ธ Enabled telemetry in workspace settings. ๐Ÿ”ง Set telemetry OTLP endpoint to http://localhost:4317. โœ… Workspace settings updated. โœ… Using Google Cloud Project ID: foo-bar ๐Ÿ”‘ Please ensure you are authenticated with Google Cloud: - Run `gcloud auth application-default login` OR ensure `GOOGLE_APPLICATION_CREDENTIALS` environment variable points to a valid service account key. - The account needs "Cloud Trace Agent", "Monitoring Metric Writer", and "Logs Writer" roles. โœ… otelcol-contrib already exists at /Users/jerop/github/gemini-cli/.gemini/otel/bin/otelcol-contrib ๐Ÿงน Cleaning up old processes and logs... โœ… Deleted old GCP collector log. ๐Ÿ“„ Wrote OTEL collector config to /Users/jerop/github/gemini-cli/.gemini/otel/collector-gcp.yaml ๐Ÿš€ Starting OTEL collector for GCP... Logs: /Users/jerop/github/gemini-cli/.gemini/otel/collector-gcp.log โณ Waiting for OTEL collector to start (PID: 65145)... โœ… OTEL collector started successfully on port 4317. โœจ Local OTEL collector for GCP is running. ๐Ÿ“„ Collector logs are being written to: /Users/jerop/github/gemini-cli/.gemini/otel/collector-gcp.log ๐Ÿ“Š View your telemetry data in Google Cloud Console: - Traces: https://console.cloud.google.com/traces/list?project=foo-bar - Metrics: https://console.cloud.google.com/monitoring/metrics-explorer?project=foo-bar - Logs: https://console.cloud.google.com/logs/query;query=logName%3D%22projects%2Ffoo-bar%2Flogs%2Fgemini_cli%22?project=foo-bar Press Ctrl+C to exit. ^C ๐Ÿ‘‹ Shutting down... โš™๏ธ Disabled telemetry in workspace settings. ๐Ÿ”ง Cleared telemetry OTLP endpoint. โœ… Workspace settings updated. ๐Ÿ›‘ Stopping otelcol-contrib (PID: 65145)... โœ… otelcol-contrib stopped. ```
Diffstat (limited to 'docs/core')
-rw-r--r--docs/core/telemetry.md235
1 files changed, 51 insertions, 184 deletions
diff --git a/docs/core/telemetry.md b/docs/core/telemetry.md
index 2252e809..b0a3d4ae 100644
--- a/docs/core/telemetry.md
+++ b/docs/core/telemetry.md
@@ -6,6 +6,32 @@ This entire system is built on the **[OpenTelemetry] (OTEL)** standard, allowing
[OpenTelemetry]: https://opentelemetry.io/
+## Quick Start
+
+### Telemetry with Google Cloud
+
+1. **Ensure Prerequisites:**
+ Ensure that:
+ - You have set the `GOOGLE_CLOUD_PROJECT` environment variable.
+ - You have authenticated with Google Cloud and have the necessary IAM roles.
+ For full details, see the [Google Cloud](#google-cloud) prerequisites.
+2. **Run the Script:** Execute the following command from the project root:
+ ```bash
+ ./scripts/telemetry_gcp.js
+ ```
+3. **View Data:** The script will provide links to view your telemetry data (traces, metrics, logs) in the Google Cloud Console.
+4. **Details:** Refer to documentation for telemetry in [Google Cloud](#google-cloud).
+
+### Local Telemetry with Jaeger UI (for Traces)
+
+1. **Run the Script:** Execute the following command from the project root:
+ ```bash
+ ./scripts/local_telemetry.js
+ ```
+2. **View Logs/Metrics:** Check the `.gemini/otel/collector.log` file for raw logs and metrics.
+3. **View Traces:** Open your browser and go to `http://localhost:16686` to see traces in the Jaeger UI.
+4. **Details:** Refer to documentation for telemetry in [Local](#local).
+
## Enabling Telemetry
You can enable telemetry in multiple ways. [Configuration](configuration.md) is primarily managed via the `.gemini/settings.json` file and environment variables, but CLI flags can override these settings for a specific session.
@@ -50,10 +76,9 @@ Learn more about OTEL exporter standard configuration in [documentation][otel-co
mkdir .gemini/otel
```
-### Local (Automated Script)
+### Local
-For the most straightforward local setup, use the `scripts/local_telemetry.js` script. This script automates the entire process of setting up a local telemetry pipeline, including configuring the necessary settings in your `.gemini/settings.json` file.
-The script installs `otelcol-contrib` (The OpenTelemetry Collector) and `jaeger` (The Jaeger UI for viewing traces). To use it:
+Use the `scripts/local_telemetry.js` script that automates the entire process of setting up a local telemetry pipeline, including configuring the necessary settings in your `.gemini/settings.json` file. The script installs `otelcol-contrib` (The OpenTelemetry Collector) and `jaeger` (The Jaeger UI for viewing traces). To use it:
1. **Run the Script**:
Execute the script from the root of the repository:
@@ -81,202 +106,44 @@ The script installs `otelcol-contrib` (The OpenTelemetry Collector) and `jaeger`
4. **Stop the Services**:
Press `Ctrl+C` in the terminal where the script is running to stop the OTEL Collector and Jaeger services.
-### Local (Manual Setup)
-
-**1. Create a Configuration File**
-
-Create the file `.gemini/otel/collector-local.yaml` with the following:
-
-```bash
-cat <<EOF > .gemini/otel/collector-local.yaml
-receivers:
- otlp:
- protocols:
- grpc:
- endpoint: "0.0.0.0:4317"
-
-processors:
- batch:
- timeout: 1s
-
-exporters:
- debug:
- verbosity: detailed
-
-service:
- telemetry:
- logs:
- level: "debug"
- pipelines:
- traces:
- receivers: [otlp]
- processors: [batch]
- exporters: [debug]
- metrics:
- receivers: [otlp]
- processors: [batch]
- exporters: [debug]
- logs:
- receivers: [otlp]
- processors: [batch]
- exporters: [debug]
-EOF
-```
-
-**2. Run the Collector**
-
-You can run the collector using `docker` or using the `otelcol-contrib` binary directly.
-
-**_Option 1: Use Docker_**
-
-This is the simplest method if you have Docker installed.
-
-1. **Run the Collector**:
-
- ```bash
- docker run --rm --name otel-collector-local \
- -p 4317:4317 \
- -v "$(pwd)/.gemini/otel/collector-local.yaml":/etc/otelcol-contrib/config.yaml \
- otel/opentelemetry-collector-contrib:latest
- ```
-
-2. **Stop the Collector**:
- ```bash
- docker stop otel-collector-local
- ```
-
-**_Option 2: Use `otelcol-contrib`_**
-
-Use this method if you prefer not to use Docker.
-
-1. **Run the Collector**:
- Once installed, run the collector with the configuration file you created earlier:
-
- ```bash
- ./otelcol-contrib --config="$(pwd)/.gemini/otel/collector-local.yaml"
- ```
-
-2. **Stop the Collector**:
- Press `Ctrl+C` in the terminal where the collector is running.
-
### Google Cloud
-This setup sends all telemetry to Google Cloud for robust, long-term analysis.
-
-**1. Prerequisites**
-
-- A Google Cloud Project ID.
-- **APIs Enabled**: Cloud Trace, Cloud Monitoring, Cloud Logging.
-- **Authentication**: A Service Account with the roles `Cloud Trace Agent`, `Monitoring Metric Writer`, and `Logs Writer`. Ensure your environment is authenticated (e.g., via `gcloud auth application-default login` or a service account key file).
-
-**2. Set environment variables**
-
-Set the `GOOGLE_CLOUD_PROJECT`, `GOOGLE_CLOUD_LOCATION`, and `GOOGLE_GENAI_USE_VERTEXAI` environment variables:
-
-```bash
-GOOGLE_CLOUD_PROJECT="YOUR_PROJECT_ID"
-GOOGLE_CLOUD_LOCATION="YOUR_PROJECT_LOCATION" # e.g., us-central1
-GOOGLE_GENAI_USE_VERTEXAI=true
-```
-
-**3. Create a Configuration File**
-
-Create `.gemini/otel/collector-gcp.yaml`:
-
-```bash
-cat <<EOF > .gemini/otel/collector-gcp.yaml
-receivers:
- otlp:
- protocols:
- grpc:
- endpoint: "0.0.0.0:4317"
+For a streamlined setup targeting Google Cloud, use the `scripts/telemetry_gcp.js` script which automates setting up a local OpenTelemetry collector that forwards data to your Google Cloud project.
-processors:
- batch:
- timeout: 1s
+1. **Prerequisites**:
-exporters:
- googlecloud:
- project: "${GOOGLE_CLOUD_PROJECT}"
- metric:
- prefix: "custom.googleapis.com/gemini_cli"
- log:
- default_log_name: "gemini_cli"
- debug:
- verbosity: detailed
+ - Ensure you have a Google Cloud Project ID.
+ - Set the `GOOGLE_CLOUD_PROJECT` environment variable to your project ID.
+ - Authenticate with Google Cloud (e.g., run `gcloud auth application-default login` or ensure `GOOGLE_APPLICATION_CREDENTIALS` is set).
+ - Ensure your account/service account has the necessary roles: "Cloud Trace Agent", "Monitoring Metric Writer", and "Logs Writer".
-service:
- pipelines:
- traces:
- receivers: [otlp]
- exporters: [googlecloud]
- metrics:
- receivers: [otlp]
- exporters: [googlecloud]
- logs:
- receivers: [otlp]
- exporters: [googlecloud]
-EOF
-```
-
-**4. Run the Collector**
-
-You can run the collector for Google Cloud using either Docker or a locally installed `otelcol` binary.
-
-**_Option 1: Use Docker _**
-
-This method encapsulates the collector and its dependencies within a container.
-
-1. **Run the Collector**:
- Choose the command that matches your authentication method.
-
- - **If using Application Default Credentials (`gcloud auth application-default login`)**:
-
- ```bash
- docker run --rm --name otel-collector-gcp \
- -p 4317:4317 \
- --user "$(id -u):$(id -g)" \
- -v "$HOME/.config/gcloud/application_default_credentials.json":/etc/gcp/credentials.json:ro \
- -e "GOOGLE_APPLICATION_CREDENTIALS=/etc/gcp/credentials.json" \
- -v "$(pwd)/.gemini/otel/collector-gcp.yaml":/etc/otelcol-contrib/config.yaml \
- otel/opentelemetry-collector-contrib:latest --config /etc/otelcol-contrib/config.yaml
- ```
-
- - **If using a Service Account Key File**:
- ```bash
- docker run --rm --name otel-collector-gcp \
- -p 4317:4317 \
- -v "/path/to/your/sa-key.json":/etc/gcp/sa-key.json:ro \
- -e "GOOGLE_APPLICATION_CREDENTIALS=/etc/gcp/sa-key.json" \
- -v "$(pwd)/.gemini/otel/collector-gcp.yaml":/etc/otelcol-contrib/config.yaml \
- otel/opentelemetry-collector-contrib:latest --config /etc/otelcol-contrib/config.yaml
- ```
-
-2. **Check Status**:
- Your telemetry data will now appear in Google Cloud Trace, Monitoring, and Logging.
+2. **Run the Script**:
+ Execute the script from the root of the repository:
-3. **Stop the Collector**:
```bash
- docker stop otel-collector-gcp
+ ./scripts/telemetry_gcp.js
```
-**_Option 2: Use `otelcol-contrib`_**
+ The script will:
-Use this method if you prefer not to use Docker.
+ - Download the `otelcol-contrib` binary if needed.
+ - Start an OTEL collector configured to receive data from the Gemini CLI and export it to your specified Google Cloud project.
+ - Automatically enable telemetry and disable sandbox mode in your workspace settings (`.gemini/settings.json`).
+ - Provide direct links to view traces, metrics, and logs in your Google Cloud Console.
+ - On exit (Ctrl+C), it will attempt to restore your original telemetry and sandbox settings.
-1. **Run the Collector**:
+3. **View Telemetry in Google Cloud**:
+ Use the links provided by the script to navigate to the Google Cloud Console and view your traces, metrics, and logs.
+
+4. **Inspect Local Collector Logs**:
+ The script redirects the local OTEL collector's output to `.gemini/otel/collector-gcp.log`. You can monitor this file for detailed information or troubleshooting:
```bash
- ./otelcol-contrib --config="file:$(pwd)/.gemini/otel/collector-gcp.yaml"
+ tail -f .gemini/otel/collector-gcp.log
```
-2. **Check Status**:
- Your telemetry data will now appear in Google Cloud Trace, Monitoring, and Logging.
-
-3. **Stop the Collector**:
- Press `Ctrl+C` in the terminal where the collector is running.
-
----
+5. **Stop the Service**:
+ Press `Ctrl+C` in the terminal where the script is running to stop the OTEL Collector.
## Data Reference: Logs & Metrics