initial playwright setup and test (#1039)
* initial playwright setup and test * try verbose logging * double check resolution * double check token * remove logs * try running on ubuntu and macos * move e2e tests * vitest ignores playwright tests * add a series of tests * tweak yarn setup * typo * update fmt * action typo * rust toolchain? * remove .only from playwright test * A snapshot a day keeps the bugs away! 📷🐛 * A snapshot a day keeps the bugs away! 📷🐛 * add pan and zoom back * try puting os in commit message * A snapshot a day keeps the bugs away! 📷🐛 .os * A snapshot a day keeps the bugs away! 📷🐛 .os * fix commit message * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-latest) * add command logs to UI for axis tests * typo * fmt * remove .only * add auto complete test * remove .only * update queries to be more playwright recommended * move test utils * remove waits from first test * remove old snapshots * re-arrange files and tests * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest) * trigger CI * refactor plane test * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest) * remove .only * try longer wait * try snap shoting * fmt * better CI names * fix * fix linux * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest) * try make executes on load a bit more robust * fix * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest) * stabilise snapshots * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest) * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-latest) * make tests run linearly * update naming * tidy * .only * update readme * trigger CI * add set tool * util clean up * update readme * readme tweaks * self-review clean up * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu) * tweak * update readme * Ensure Vite preview server is running before running any Playwright tests (#1114) * Ensure that Vite is serving before tests run * Don't break secrets if developer has extra blank line in env file * @mxschmitt's suggestions * fix * add wait-on types * tsconfig fix * update readme * code template for sktech on each plane test --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Frank Noirot <frank@kittycad.io>
This commit is contained in:
Kurt Hutten
GitHub
107
README.md
107
README.md
@ -176,3 +176,110 @@ $ cargo +nightly fuzz run parser
|
||||
|
||||
For more information on fuzzing you can check out
|
||||
[this guide](https://rust-fuzz.github.io/book/cargo-fuzz.html).
|
||||
|
||||
|
||||
### Playwright
|
||||
|
||||
First time running plawright locally, you'll need to add the secrets file
|
||||
```bash
|
||||
touch ./e2e/playwright/playwright-secrets.env
|
||||
echo 'token="your-token"' > ./e2e/playwright/playwright-secrets.env
|
||||
```
|
||||
then replace "your-token" with a dev token from dev.kittycad.io/account/api-tokens
|
||||
|
||||
then:
|
||||
run playwright
|
||||
```
|
||||
yarn playwright test
|
||||
```
|
||||
|
||||
run a specific test suite
|
||||
```
|
||||
yarn playwright test src/e2e-tests/example.spec.ts
|
||||
```
|
||||
|
||||
run a specific test change the test from `test('...` to `test.only('...`
|
||||
(note if you commit this, the tests will instantly fail without running any of the tests)
|
||||
|
||||
run headed
|
||||
```
|
||||
yarn playwright test --headed
|
||||
```
|
||||
|
||||
run with step through debugger
|
||||
```
|
||||
PWDEBUG=1 yarn playwright test
|
||||
```
|
||||
However, if you want a debugger I recommend using VSCode and the `playwright` extension, as the above command is a cruder debugger that steps into every function call which is annoying.
|
||||
With the extension you can set a breakpoint after `waitForDefaultPlanesVisibilityChange` in order to skip app loading, then the vscode debugger's "step over" is much better for being able to stay at the right level of abstraction as you debug the code.
|
||||
|
||||
If you want to limit to a single browser use `--project="webkit"` or `firefox`, `Google Chrome`
|
||||
Or comment out browsers in `playwright.config.ts`.
|
||||
|
||||
note chromium has encoder compat issues which is why were testing against the branded 'Google Chrome'
|
||||
|
||||
You may consider using the VSCode extension, it's useful for running individual threads, but some some reason the "record a test" is locked to chromium with we can't use. A work around is to us the CI `yarn playwright codegen -b wk --load-storage ./store localhost:3000`
|
||||
|
||||
<details>
|
||||
<summary>
|
||||
|
||||
Where `./store` should look like this
|
||||
|
||||
</summary>
|
||||
|
||||
```JSON
|
||||
{
|
||||
"cookies": [],
|
||||
"origins": [
|
||||
{
|
||||
"origin": "http://localhost:3000",
|
||||
"localStorage": [
|
||||
{
|
||||
"name": "store",
|
||||
"value": "{\"state\":{\"openPanes\":[\"code\"]},\"version\":0}"
|
||||
},
|
||||
{
|
||||
"name": "persistCode",
|
||||
"value": ""
|
||||
},
|
||||
{
|
||||
"name": "TOKEN_PERSIST_KEY",
|
||||
"value": "your-token"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
However because much of our tests involve clicking in the stream at specific locations, it's code-gen looks `await page.locator('video').click();` when really we need to use a pixel coord, so I think it's of limited use.
|
||||
|
||||
#### Some notes on CI
|
||||
|
||||
The tests are broken into snapshot tests and non-snapshot tests, and they run in that order, they automatically commit new snap shots, so if you see an image commit check it was an intended change. If we have non-determinism in the snapshots such that they are always committing new images, hopefully this annoyance makes us fix them asap, if you notice this happening let Kurt know. But for the odd occasion `git reset --hard HEAD~ && git push -f` is your friend.
|
||||
|
||||
How to interpret failing playwright tests?
|
||||
If your tests fail, click through to the action and see that the tests failed on a line that includes `await page.getByTestId('loading').waitFor({ state: 'detached' })`, this means the test fail because the stream never started. It's you choice if you want to re-run the test, or ignore the failure.
|
||||
|
||||
We run on ubuntu and macos, because safari doesn't work on linux because of the dreaded "no RTCPeerConnection variable" error. But linux runs first and then macos for the same reason that we limit the number of parallel tests to 1 because we limit stream connections per user, so tests would start failing we if let them run together.
|
||||
|
||||
#### Getting started writing a playwright test in our app
|
||||
|
||||
Besides following the instructions above and using the playwright docs, our app is weird because of the whole stream thing, which means our testing is weird. Because we've just figured out this stuff and therefore docs might go stale quick here's a 15min vid/tutorial
|
||||
|
||||
https://github.com/KittyCAD/modeling-app/assets/29681384/6f5e8e85-1003-4fd9-be7f-f36ce833942d
|
||||
|
||||
<details>
|
||||
|
||||
<summary>
|
||||
Ps for the debug panel, the following JSON is useful for snapping the camera
|
||||
</summary>
|
||||
|
||||
```JSON
|
||||
{"type":"modeling_cmd_req","cmd_id":"054e5472-e5e9-4071-92d7-1ce3bac61956","cmd":{"type":"default_camera_look_at","center":{"x":15,"y":0,"z":0},"up":{"x":0,"y":0,"z":1},"vantage":{"x":30,"y":30,"z":30}}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
Reference in New Issue
Block a user