Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

@suite/desktop-app and @suite/web-app e2e tests

@suite/e2e uses Playwright to run e2e tests. It also uses trezor-user-env which is daily built into a docker image providing all the necessary instrumentation required to run tests (bridge and emulators).

Run it locally

Note: All paths below are relative to the root of trezor-suite repository, if not specified otherwise.

Prerequisites

Common steps

  1. macOS only: Run Docker and go to Preferences -> Resources -> Advanced and increase RAM to at least 4GB. Otherwise, the app during tests does not even load.
  2. In terminal window, navigate to trezor-user-env repo root and run ./run.sh.
  3. In another terminal window, run yarn workspace @trezor/suite-e2e docker:suite-sync to have local Relay server instance
  4. In workspace @trezor/suite-e2e create a .env file according to the .example.env

Web

  1. In another terminal window, run web Suite with yarn suite:dev.
  2. In another terminal window, run yarn workspace @trezor/suite-e2e test:e2e:web.

Desktop

  1. TEST_BUILD=true yarn workspace @suite/desktop-app build:ui

    Produces desktop-app/build directory with javascript bundles & assets in production mode for the electron-renderer process. TEST_BUILD env variable serves to mock bundled message-system config .

    Note: This step needs to be repeated on each change in suite or suite-desktop-ui package.

  2. yarn workspace @suite/desktop-app build:app

    Produces desktop-app/dist directory with javascript bundles & assets in production mode for the electron-main process.

    Note: This step needs to be repeated on each change in connect or suite-desktop-core package.

  3. yarn workspace @trezor/suite-e2e test:e2e:desktop

Troubleshooting

  1. To run tests headed (showing UI) you can add: --headed.

  2. To run just one test file you can do: yarn workspace @trezor/suite-e2e test:e2e:web general/wallet-discovery.test.ts or yarn workspace @trezor/suite-e2e test:e2e:desktop general/wallet-discovery.test.ts

  3. To run just one test you can add: -g "Basic cardano walkthrough"

  4. To run on a specific device model you can only run the device specific project i.e.: yarn workspace @trezor/suite-e2e test:e2e:web --project=T3T1 general/wallet-discovery.test.ts

  5. To run tests with canary FW you need to use canary yarn script i.e. yarn workspace @trezor/suite-e2e test:e2e:web:canary general/wallet-discovery.test.ts

  6. To open advance playwright runner/debugger ui you can add: --ui

  7. To check for flakiness you can specify test/suite and how many time it should run: --repeat-each=10

  8. To check for flakiness on CI you can edit in packages/suite/package.json script "test:orchestrated:e2e:desktop": "NODE_OPTIONS='--no-warnings=DEP0040' yarn xvfb-maybe -- pwc-p --config=./playwright-config/playwright-desktop-nightly.config.ts --repeat-each=30 --grep=<test-file-name>",, commit, push and run this CI against your branch. This will run this one test 30 times. Please, always specify limited number of tests, never run full suite 30 times.

  9. To debug test add await page.pause(); to place where you want test to stop. Debugger window will open. This works only in --headed run.

  10. To enable Debug Tools in the browser press Ctrl+Shift+I

  11. To enable Electron verbose logging add env variable LOGLEVEL=debug or any other level

  12. To increase test timeouts when your local run exceed 1m limit, you can specify test timeout override in packages/suite/.env. (UI runner –ui needs to be restarted to reflect the change in .env)

  13. To find a breaking commit in develop you can checkout latest develop and run yarn workspace @suite/desktop-app git:bisect <last_good_commit> <desktop|web> <test_file>

Contribution

Please follow our general Playwright contribution guide

Tags

Each test must be assigned a tag according to what device model it is supported by the test i.e.

  • @T1B1
  • @T2T1
  • @T3B1
  • @T3T1
  • @T3W1
  • @noDevice

At the moment, there are these additional tags:

  • @desktopOnly
  • @webOnly
  • @skipOnPR
  • @optional
  • @specificFirmware

Device coverage on PR

Most E2E tests are specified to cover T3T1 and T3W1, the two flagship models. But to save Currents quota, T3T1 is skipped on PR test runs, and only T3W1 is covered, unless the test is exclusive (T3T1-only) test. Meanwhile, nightly tests provide full coverage, i.e. they run both flagship models. This is configured entirely in the Playwright project configs. To summarize:

  • in PR run, if a test specifies T3T1 alongside any other models, T3T1 is skipped, and only other models are covered (usually just T3W1)
  • in PR run, if a test specifies T3T1 as its only model, it is covered
  • in nightly run, all models specified in the test are covered

@desktopOnly or @webOnly

Some tests are only applicable for Desktop app or Web and you can use this tag to notify the runner, that the test should be ignored when running against opposite Suite. This negative filtering is done on playwright-config level.

Currently, we are also applying @webOnly as a positive filter on Web PR runs. This is done in Web PR workflow definition. Meaning, Web PR runs execute only tests with @WebOnly tag to reduce amount of test run daily and save quota on Currents, where we are paying extra for any test runs over 100 000.

@skipOnPR

Tests that must never run on a PR, not even when the LLM test selector or an edited test file targets them. Typical reasons are nonce collisions when two runs overlap, or old app versions that only the deployed instances provide. The PR Playwright configs, including the spec-list (-pr-all) ones, filter this tag out. These tests run on nightly, canary and release runs.

@optional

Tests that are excluded from the full PR run but run on a PR when the LLM test selector recommends them or the PR edits the test file itself. Use it for low-priority slow tests or tests that depend on a flaky backend, and, when the reason is not obvious from the test itself, state it in a one-line comment above the describe. Only the full-run PR configs filter this tag out; nightly, canary, release and spec-list runs include it.

@specificFirmware

Some tests must run on specific Firmware version. That version is setup and defined in test. This tag lets our runner know, that this test should not be included in Canary nightly run.

Results

Results contains traces, metadata, logs, screenshots, videos and various useful information for debugging. Traces contain electron logs of our desktop suit app. Both in CI, currents and local env (suite/e2e/test-results). CI runs have artifact with logs from trezor-user-env. They exist per group. Log are separated by Log entry - - - STARTING TEST trading/swap-tokens.test.ts and - - - FINISHING TEST trading/swap-tokens.test.ts

  • electrum-regtest.txt
  • trezor-user-env-debugging.log
  • tenv-emulator-bridge-debugging.log
  • trezor-user-env-version.txt
  • quota-db.txt
  • suite-sync.txt

Local Relay Server

The local relay server is used by the Suite Sync automation and its test suite to provide an isolated, controllable endpoint and deterministic test data. Each test start by restarting the server and wiping its data store, so do not expect data to persist between test runs. After a test run finishes, the data remains on the server for troubleshooting until the next setup wipe. The server runs from a Docker image pulled from a tagged image in our GitHub repository. Those images are build as part of Suite Sync server pipelines

  • Relay: http://localhost:4000
  • Quota manager: http://localhost:4001
  • Health check: http://localhost:4002

Currents.dev

Test reports are uploaded to currents.dev