@trezor/suite-desktop and @trezor/suite-web e2e tests

@trezor/suite uses Playwright to run e2e tests. It also uses trezor-user-env which is daily built into a docker image providing all the necessary instrumentation required to run tests (bridge and emulators).

Run it locally

Note: All paths below are relative to the root of trezor-suite repository, if not specified otherwise.

Common

Full steps:
(in case of Linux with X11 support, skip to step 6.)

  1. Run XQuartz. Wait till it is launched. Leave it running in the background.
  2. In XQuartz settings go to Preferences -> Security and enable "Allow connections from network clients".
  3. Open a new terminal window (not in XQuartz) and add yourself to the X access control list:
    • xhost +127.0.0.1
    • You will probably need to logout/login after XQuartz installation to have xhost command available.
  4. Run Docker and go to Preferences -> Resources -> Advanced and increase RAM to at least 4GB. Otherwise, the app during tests does not even load.
  5. In the terminal window, set two environment variables:
    • export HOSTNAME=`hostname`
    • export DISPLAY=:0
  6. In terminal window, navigate to trezor-user-env repo root and run ./run.sh.
  7. In workspace packages/suite-desktop-core create a .env file according to the .example.env

Web

  1. In another window, run web Suite with yarn suite:dev.
  2. In a third window, run yarn workspace @trezor/suite-desktop-core test:e2e:web.

Desktop

  1. yarn workspace @trezor/suite-desktop build:ui

    Produces suite-desktop/build directory with javascript bundles & assets in production mode for the electron-renderer process.

    Note: This step needs to be repeated on each change in suite or suite-desktop-ui package.

  2. yarn workspace @trezor/suite-desktop build:app

    Produces suite-desktop/dist directory with javascript bundles & assets in production mode for the electron-main process.

    Note: This step needs to be repeated on each change in connect or suite-desktop-core package.

  3. yarn workspace @trezor/suite-desktop-core test:e2e:desktop

Troubleshooting

  1. To run tests headed (showing UI) you can add: --headed.

  2. To run just one test file you can do: yarn workspace @trezor/suite-desktop-core test:e2e general/wallet-discovery.test.ts

  3. To run just one test you can add: -g "Basic cardano walkthrough"

  4. To run one group you can add: --grep @group=wallet

  5. To open advance playwright runner/debugger ui you can add: --ui

  6. To check for flakiness you can specify test/suite and how many time it should run: --repeat-each=10

  7. To check for flakiness on CI you can edit in packages/suite-desktop-core/package.json script "test:orchestrated:e2e:desktop": "NODE_OPTIONS='--no-warnings=DEP0040' yarn xvfb-maybe -- pwc-p --config=./e2e/playwright.config.ts --project=desktop --repeat-each=30 <test-file-name>",, commit, push and run this CI against your branch. This will run this one test 30 times. Please, always specify limited number of tests, never run full suite 30 times.

  8. To debug test add await page.pause(); to place where you want test to stop. Debugger window will open. This works only in --headed run.

  9. To enable Debug Tools in the browser press Ctrl+Shift+I

  10. To enable Electron verbose logging add env variable LOGLEVEL=debug or any other level

  11. To increase test timeouts when your local run exceed 1m limit, you can specify test timeout override in packages/suite-desktop-core/.env. (UI runner --ui needs to be restarted to reflect the change in .env)

  12. To run with x-main firmware instead of x-latest you can use CANARY_FIRMWARE env variable like this: CANARY_FIRMWARE=true yarn workspace @trezor/suite-desktop-core test:e2e:web or CANARY_FIRMWARE=true yarn workspace @trezor/suite-desktop-core test:e2e:desktop

  13. To find a breaking commit in develop you can checkout latest develop and run yarn workspace @trezor/suite-desktop git:bisect <last_good_commit> <desktop|web> <test_file>

Contribution

Please follow our general Playwright contribution guide

Tags

Each test should be assigned a tag

At the moment, there are the following tags:

  • @group=[string]
  • @desktopOnly
  • @webOnly
  • @nightlyOnly
  • @snapshot

@group

Assigning a @group allows filtering based on logical test category

  • @group=wallet
  • @group=suite
  • @group=device-management
  • @group=other
  • @group=metadata
  • @group=passphrase
  • @group=settings
  • @group=trading

@desktopOnly or @webOnly

Some tests are only applicable for Desktop app or Web and you can use this tag to notify the runner, that the test should be ignored when running against opposite Suite. This negative filtering is done on playwright-config level.

Currently, we are also applying @webOnly as a positive filter on Web PR runs. This is done in Web PR workflow definition. Meaning, Web PR runs execute only tests with @WebOnly tag to reduce amount of test run daily and save quota on Currents, where we are paying extra for any test runs over 100 000.

@nightlyOnly

Some tests should run on nighty runs only for specific reasons. This tags is used for reversed filtering in PR and release workflows definitions. This means tests with this tags will run only on nightly and canary runs.

@snapshot

Some tests are using visual regression comparison by comparing specific element with a prerecorded snapshot or by comparing Aria snapshot. This tag serve for easier updating of snapshots and monitoring.

Updating snapshots

Changes in implementation or environment may demand updating our Aria snapshots or prerecorded image snapshots that serve for visual regression comparison. To do so:

  1. Run yarn test:e2e:update-snapshots, this will run all tests with @snapshot tag and record new snapshots if there is difference.
    • Alternatively, you can run yarn test:e2e <filter one specific test or test file> --update-snapshots to run even smaller amount of tests and to do a update just for that test or test file.
  2. Once these tests finish, you will be presented with results which may contain diff patch files of Aria snapshots like in this example:
$ yarn test:e2e ios --update-snapshots

Running 1 test using 1 worker

  ✓  1 …wser › Suite does not support iOS @group=other @webOnly @snapshot (5.7s)

New baselines created for:

  e2e/tests/browser/ios.test.ts

  git apply e2e/test-results/rebaselines.patch

  1 passed (6.4s)

To open last HTML report run:

  yarn playwright show-report
  1. Apply all patches and review that aria snapshot changes. In above example it would be git apply e2e/test-results/rebaselines.patch
  2. Review all new image snapshots that were generated by the first step. They are automatically stored in packages/suite-desktop-core/e2e/snapshots folder and should be visible in your git status.
  3. Add, commit, and create PR

Results

Results contains traces, metadata, logs, screenshots, videos and various useful information for debugging. Traces contain electron logs of our desktop suit app. Both in CI, currents and local env (packages/suite-desktop-core/e2e/test-results). CI runs have artifact with logs from trezor-user-env. They exist per group. Log are separated by Log entry - - - STARTING TEST trading/swap-tokens.test.ts and - - - FINISHING TEST trading/swap-tokens.test.ts

  • electrum-regtest.txt
  • trezor-user-env-debugging.log
  • tenv-emulator-bridge-debugging.log

Currents.dev

Test reports are uploaded to currents.dev