summaryrefslogtreecommitdiff
path: root/docs/integration-tests.md
diff options
context:
space:
mode:
Diffstat (limited to 'docs/integration-tests.md')
-rw-r--r--docs/integration-tests.md149
1 files changed, 149 insertions, 0 deletions
diff --git a/docs/integration-tests.md b/docs/integration-tests.md
new file mode 100644
index 00000000..d86a663f
--- /dev/null
+++ b/docs/integration-tests.md
@@ -0,0 +1,149 @@
+# Integration Tests
+
+This document provides a detailed overview of the integration testing framework used in this project.
+
+## Overview
+
+The integration tests are designed to validate the end-to-end functionality of the Gemini CLI. They execute the built binary in a controlled environment and verify that it behaves as expected when interacting with the file system.
+
+These tests are located in the `integration-tests` directory and are run using a custom test runner that provides a consistent and configurable testing environment.
+
+## Running the Tests
+
+The integration tests are not run as part of the default `npm run test` command. They must be run explicitly using the `npm run test:integration:sandbox:none` script.
+
+Also as a developer for full context a shortcut can be found at
+
+```bash
+npm run test:e2e
+```
+
+## Running a specfic set of tests
+
+To run a 1 or more test files you can use `npm run <integration test command> <file_name1> ....` where <integration test command> is any of `test:e2e` or `test:integration*` and <file_name> is any of the files in `integration/<file_name>.test.js`
+
+```bash
+npm run test:e2e write_file
+```
+
+### Running a Single Test by Name
+
+To run a single test by its name, use the `--test-name-pattern` flag:
+
+```bash
+npm run test:e2e -- --test-name-pattern "reads a file"
+```
+
+### Running All Tests
+
+To run the entire suite of integration tests, use the following command:
+
+```bash
+npm run test:integration:all
+```
+
+### Sandbox Matrix
+
+The `all` command will run tests for `no sandboxing`, `docker` and `podman`.
+Each individual type can be run as
+
+```bash
+npm run test:integration:all
+```
+
+```bash
+npm run test:integration:sandbox-none
+```
+
+```bash
+npm run test:integration:sandbox:docker
+```
+
+```bash
+npm run test:integration:sandbox:podman
+```
+
+## Diagnostics
+
+The integration test runner provides several options for diagnostics to help track down test failures.
+
+### Keeping Test Output
+
+You can preserve the temporary files created during a test run for inspection. This is useful for debugging issues with file system operations.
+
+To keep the test output, you can either use the `--keep-output` flag or set the `KEEP_OUTPUT` environment variable to `true`.
+
+```bash
+# Using the flag
+npm run test:integration:sandbox:none -- --keep-output
+
+# Using the environment variable
+KEEP_OUTPUT=true npm run test:integration:sandbox:none
+```
+
+When output is kept, the test runner will print the path to the unique directory for the test run.
+
+### Verbose Output
+
+For more detailed debugging, the `--verbose` flag will stream the real-time output from the `gemini` command to the console. This is useful for observing the command's behavior as it runs.
+
+```bash
+npm run test:integration:sandbox:none -- --verbose
+```
+
+When using `--verbose` with `--keep-output`, the output is streamed to the console and also saved to a log file within the test's temporary directory.
+
+The verbose output is formatted to clearly identify the source of the logs:
+
+```
+--- TEST: <file-name-without-js>:<test-name> ---
+... output from the gemini command ...
+--- END TEST: <file-name-without-js>:<test-name> ---
+```
+
+## Linting and Formatting
+
+To ensure code quality and consistency, the integration test files are linted as part of the main build process. You can also manually run the linter and auto-fixer.
+
+### Running the Linter
+
+To check for linting errors, run the following command:
+
+```bash
+npm run lint
+```
+
+### Automatically Fixing Issues
+
+To automatically fix any fixable linting errors, run:
+
+```bash
+npm run lint --fix
+```
+
+## Directory Structure
+
+The integration tests create a unique directory for each test run inside the `.integration-tests` directory. Within this directory, a subdirectory is created for each test file, and within that, a subdirectory is created for each individual test case.
+
+This structure makes it easy to locate the artifacts for a specific test run, file, or case.
+
+```
+.integration-tests/
+└── <run-id>/
+ └── <test-file-name>.test.js/
+ └── <test-case-name>/
+ ├── output.log
+ └── ...other test artifacts...
+```
+
+## Continuous Integration
+
+To ensure the integration tests are always run, a GitHub Actions workflow is defined in `.github/workflows/e2e.yml`. This workflow automatically runs the integration tests on every pull request and push to the `main` branch.
+
+The workflow uses a matrix strategy to run the tests in different sandboxing environments:
+
+- `sandbox:none`: Runs the tests without any sandboxing.
+- `sandbox:docker`: Runs the tests in a Docker container.
+- `sandbox:podman`: Runs the tests in a Podman container.
+
+This ensures that the Gemini CLI is tested across a variety of environments, improving its robustness and reliability.