Move all tests over to electron (#4484)
* Move all tests over to electron * Pass the correct param to playwright-electron.sh * Add shebang to script and add macos-14-large as a target * Get sketch-tests.spec.ts passing in electron * Try out 4 workers * Got testing-segment-overlays passing * Pass testing-selections.spec.ts * Go back to fix up sketch-tests test * Pass various.spec.ts, by far the hardest one * Pass can-sketch-on-all-planes... with ease * Pass command bar tests * fmt * Completely fix code mirror text navigating for tests * Pass debug pane tests * Pass desktop export tests * Pass editor tests * Pass file tree tests * Pass onboarding tests * Corrected a fixme in file-tree.spec! * Painfully fix hardcoded coordinates in point-click.spec * Pass machine.spec tests * Pass projects, fought hard with filechooser * Pass regresion-tests.spec tests * Pass network and connection tests * Pass camera-movement.spec tests * Extreme time eaten by gizmo test fixes. All passing now. * Merge main (tests changed x_x) and pass all constraints.spec tests (pain) * Pass another painful spec suite: testing-settings * Pass perspective-toggle, interesting note * Pass samples loading tests * Pass app header tests * Pass text-to-cad tests * Pass segment-overlays (minor ache) and ability to switch to web if needed :) * Fix a ton of syntax changes and deflake 2 more tests (pain) * Correct all tsc errors * Remove to-electron script * Add an f-ton of shit because playwright doesnt want S P R E A D * Try CI again * Stop snapshots of exports (already test in e2e) * Fix flake in double click editor * Hopefully help CI flake * Fixmes, fixmes everywhere * One more fixme to settings * Skip another code pane flake * Port jess's projects.spec tests * fixup * Reuse electron window; difficult task * Rebased and refixed * Remove duplicate cases * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest-8-cores) * Reduce the workers to something CI can handle * Lower it further, we need to think about the others * Update package.json Co-authored-by: Pierre Jacquier <pierrejacquier39@gmail.com> * Update package.json Co-authored-by: Pierre Jacquier <pierrejacquier39@gmail.com> * Fix the last tests and tsc errors * Timeout to 120 and windows-2022-16core * Fix windows runner detection, enable concurrency temporarily * Hopefully this time fix windows runner detection * Comment out Vector, add back removed camera test code * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest-8-cores) * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * Fix camera tests again * Massively deflake a whole class of tests * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest-8-cores) * Try new CI and fix small onboarding test * Derp * No github tuning * Try mac * Add back all the OS * Lord, hallow be thy name * A snapshot a day keeps the bugs away! 📷🐛 (OS: namespace-profile-ubuntu-8-cores) * A snapshot a day keeps the bugs away! 📷🐛 (OS: windows-latest-8-cores) * One last try with window-16-cores * Trigger CI * Try AWS Windows runner * Passing on windows locally with a few skips * Skip more win tests, add back all three oses * A snapshot a day keeps the bugs away! 📷🐛 (OS: windows-16-cores) * Add two more fixmes * 2 more fixmes * skip segment overlays on win32 * Another fixme * Trigger CI * Trigger CI * Quick clean up * Move all tests over to electron * Pass the correct param to playwright-electron.sh * Add shebang to script and add macos-14-large as a target * Get sketch-tests.spec.ts passing in electron * Try out 4 workers * Got testing-segment-overlays passing * Pass testing-selections.spec.ts * Go back to fix up sketch-tests test * Pass various.spec.ts, by far the hardest one * Pass can-sketch-on-all-planes... with ease * Pass command bar tests * fmt * Completely fix code mirror text navigating for tests * Pass debug pane tests * Pass desktop export tests * Pass editor tests * Pass file tree tests * Pass onboarding tests * Corrected a fixme in file-tree.spec! * Painfully fix hardcoded coordinates in point-click.spec * Pass machine.spec tests * Pass projects, fought hard with filechooser * Pass regresion-tests.spec tests * Pass network and connection tests * Pass camera-movement.spec tests * Extreme time eaten by gizmo test fixes. All passing now. * Merge main (tests changed x_x) and pass all constraints.spec tests (pain) * Pass another painful spec suite: testing-settings * Pass perspective-toggle, interesting note * Pass samples loading tests * Pass app header tests * Pass text-to-cad tests * Pass segment-overlays (minor ache) and ability to switch to web if needed :) * Fix a ton of syntax changes and deflake 2 more tests (pain) * Correct all tsc errors * Remove to-electron script * Add an f-ton of shit because playwright doesnt want S P R E A D * Try CI again * Stop snapshots of exports (already test in e2e) * Fix flake in double click editor * Hopefully help CI flake * Fixmes, fixmes everywhere * One more fixme to settings * Skip another code pane flake * Port jess's projects.spec tests * fixup * Reuse electron window; difficult task * Rebased and refixed * Remove duplicate cases * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest-8-cores) * Reduce the workers to something CI can handle * Lower it further, we need to think about the others * Update package.json Co-authored-by: Pierre Jacquier <pierrejacquier39@gmail.com> * Update package.json Co-authored-by: Pierre Jacquier <pierrejacquier39@gmail.com> * Fix the last tests and tsc errors * Timeout to 120 and windows-2022-16core * Fix windows runner detection, enable concurrency temporarily * Hopefully this time fix windows runner detection * Comment out Vector, add back removed camera test code * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest-8-cores) * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * Fix camera tests again * Massively deflake a whole class of tests * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * A snapshot a day keeps the bugs away! 📷🐛 (OS: macos-14-large) * Try new CI and fix small onboarding test * Derp * No github tuning * Try mac * Add back all the OS * Lord, hallow be thy name * A snapshot a day keeps the bugs away! 📷🐛 (OS: namespace-profile-ubuntu-8-cores) * A snapshot a day keeps the bugs away! 📷🐛 (OS: windows-latest-8-cores) * Try AWS Windows runner * Passing on windows locally with a few skips * Trigger CI * A snapshot a day keeps the bugs away! 📷🐛 (OS: namespace-profile-ubuntu-8-cores) * fmt, tsc, lint * Enable two fixmes again * Fix lint, codespell, fmt * Fix lint * Don't run e2e on draft, add back concurrency, clean up * One last windows skip --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Pierre Jacquier <pierrejacquier39@gmail.com> Co-authored-by: Pierre Jacquier <pierre@zoo.dev>
This commit is contained in:
49fl
GitHub
@ -1,22 +1,21 @@
|
||||
import {
|
||||
expect,
|
||||
Page,
|
||||
Download,
|
||||
BrowserContext,
|
||||
TestInfo,
|
||||
_electron as electron,
|
||||
ElectronApplication,
|
||||
Locator,
|
||||
test,
|
||||
} from '@playwright/test'
|
||||
import { test, Page } from './zoo-test'
|
||||
import { EngineCommand } from 'lang/std/artifactGraph'
|
||||
import fsp from 'fs/promises'
|
||||
import fsSync from 'fs'
|
||||
import { join } from 'path'
|
||||
import path from 'path'
|
||||
import pixelMatch from 'pixelmatch'
|
||||
import { PNG } from 'pngjs'
|
||||
import { Protocol } from 'playwright-core/types/protocol'
|
||||
import type { Models } from '@kittycad/lib'
|
||||
import { APP_NAME, COOKIE_NAME } from 'lib/constants'
|
||||
import { COOKIE_NAME } from 'lib/constants'
|
||||
import { secrets } from './secrets'
|
||||
import {
|
||||
TEST_SETTINGS_KEY,
|
||||
@ -30,6 +29,10 @@ import { isErrorWhitelisted } from './lib/console-error-whitelist'
|
||||
import { isArray } from 'lib/utils'
|
||||
import { reportRejection } from 'lib/trap'
|
||||
|
||||
const toNormalizedCode = (text: string) => {
|
||||
return text.replace(/\s+/g, '')
|
||||
}
|
||||
|
||||
type TestColor = [number, number, number]
|
||||
export const TEST_COLORS = {
|
||||
WHITE: [249, 249, 249] as TestColor,
|
||||
@ -98,11 +101,16 @@ async function removeCurrentCode(page: Page) {
|
||||
}
|
||||
|
||||
export async function sendCustomCmd(page: Page, cmd: EngineCommand) {
|
||||
await page.getByTestId('custom-cmd-input').fill(JSON.stringify(cmd))
|
||||
const json = JSON.stringify(cmd)
|
||||
await page.getByTestId('custom-cmd-input').fill(json)
|
||||
await expect(page.getByTestId('custom-cmd-input')).toHaveValue(json)
|
||||
await page.getByTestId('custom-cmd-send-button').scrollIntoViewIfNeeded()
|
||||
await page.getByTestId('custom-cmd-send-button').click()
|
||||
}
|
||||
|
||||
async function clearCommandLogs(page: Page) {
|
||||
await page.getByTestId('custom-cmd-input').fill('')
|
||||
await page.getByTestId('clear-commands').scrollIntoViewIfNeeded()
|
||||
await page.getByTestId('clear-commands').click()
|
||||
}
|
||||
|
||||
@ -150,6 +158,19 @@ export async function closePane(page: Page, testId: string) {
|
||||
|
||||
async function openKclCodePanel(page: Page) {
|
||||
await openPane(page, 'code-pane-button')
|
||||
|
||||
// Code Mirror lazy loads text! Wowza! Let's force-load the text for tests.
|
||||
await page.evaluate(() => {
|
||||
// editorManager is available on the window object.
|
||||
//@ts-ignore this is in an entirely different context that tsc can't see.
|
||||
editorManager._editorView.dispatch({
|
||||
selection: {
|
||||
//@ts-ignore this is in an entirely different context that tsc can't see.
|
||||
anchor: editorManager._editorView.docView.length,
|
||||
},
|
||||
scrollIntoView: true,
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
async function closeKclCodePanel(page: Page) {
|
||||
@ -165,6 +186,9 @@ async function closeKclCodePanel(page: Page) {
|
||||
|
||||
async function openDebugPanel(page: Page) {
|
||||
await openPane(page, 'debug-pane-button')
|
||||
|
||||
// The debug pane needs time to load everything.
|
||||
await page.waitForTimeout(3000)
|
||||
}
|
||||
|
||||
export async function closeDebugPanel(page: Page) {
|
||||
@ -412,6 +436,10 @@ export async function getUtils(page: Page, test_?: typeof test) {
|
||||
.boundingBox({ timeout: 5_000 })
|
||||
.then((box) => ({ ...box, x: box?.x || 0, y: box?.y || 0 })),
|
||||
codeLocator: page.locator('.cm-content'),
|
||||
crushKclCodeIntoOneLineAndThenMaybeSome: async () => {
|
||||
const code = await page.locator('.cm-content').innerText()
|
||||
return code.replaceAll(' ', '').replaceAll('\n', '')
|
||||
},
|
||||
normalisedEditorCode: async () => {
|
||||
const code = await page.locator('.cm-content').innerText()
|
||||
return normaliseKclNumbers(code)
|
||||
@ -482,13 +510,18 @@ export async function getUtils(page: Page, test_?: typeof test) {
|
||||
)
|
||||
},
|
||||
|
||||
toNormalizedCode: (text: string) => {
|
||||
return text.replace(/\s+/g, '')
|
||||
toNormalizedCode(text: string) {
|
||||
return toNormalizedCode(text)
|
||||
},
|
||||
|
||||
editorTextMatches: async (code: string) => {
|
||||
async editorTextMatches(code: string) {
|
||||
const editor = page.locator(editorSelector)
|
||||
return expect(editor).toHaveText(code, { useInnerText: true })
|
||||
return expect
|
||||
.poll(async () => {
|
||||
const text = await editor.textContent()
|
||||
return toNormalizedCode(text ?? '')
|
||||
})
|
||||
.toContain(toNormalizedCode(code))
|
||||
},
|
||||
|
||||
pasteCodeInEditor: async (code: string) => {
|
||||
@ -514,7 +547,7 @@ export async function getUtils(page: Page, test_?: typeof test) {
|
||||
page.getByRole('button', { name: 'Start Sketch' })
|
||||
).not.toBeDisabled()
|
||||
await page.getByTestId('create-file-button').click()
|
||||
await page.getByTestId('file-rename-field').fill(name)
|
||||
await page.getByTestId('tree-input-field').fill(name)
|
||||
await page.keyboard.press('Enter')
|
||||
})
|
||||
},
|
||||
@ -674,6 +707,34 @@ export const makeTemplate: (
|
||||
}
|
||||
}
|
||||
|
||||
const PLAYWRIGHT_DOWNLOAD_DIR = 'downloads-during-playwright'
|
||||
|
||||
export const getPlaywrightDownloadDir = (page: Page) => {
|
||||
return path.resolve(page.dir, PLAYWRIGHT_DOWNLOAD_DIR)
|
||||
}
|
||||
|
||||
const moveDownloadedFileTo = async (page: Page, toLocation: string) => {
|
||||
await fsp.mkdir(path.dirname(toLocation), { recursive: true })
|
||||
|
||||
const downloadDir = getPlaywrightDownloadDir(page)
|
||||
|
||||
// Expect there to be at least one file
|
||||
await expect
|
||||
.poll(async () => {
|
||||
const files = await fsp.readdir(downloadDir)
|
||||
return files.length
|
||||
})
|
||||
.toBeGreaterThan(0)
|
||||
|
||||
// Go through the downloads dir and move files to new location
|
||||
const files = await fsp.readdir(downloadDir)
|
||||
|
||||
// Assumption: only ever one file here.
|
||||
for (let file of files) {
|
||||
await fsp.rename(path.resolve(downloadDir, file), toLocation)
|
||||
}
|
||||
}
|
||||
|
||||
export interface Paths {
|
||||
modelPath: string
|
||||
imagePath: string
|
||||
@ -686,7 +747,8 @@ export const doExport = async (
|
||||
exportFrom: 'dropdown' | 'sidebarButton' | 'commandBar' = 'dropdown'
|
||||
): Promise<Paths> => {
|
||||
if (exportFrom === 'dropdown') {
|
||||
await page.getByRole('button', { name: APP_NAME }).click()
|
||||
await page.getByTestId('project-sidebar-toggle').click()
|
||||
|
||||
const exportMenuButton = page.getByRole('button', {
|
||||
name: 'Export current part',
|
||||
})
|
||||
@ -727,25 +789,12 @@ export const doExport = async (
|
||||
}
|
||||
await expect(page.getByText('Confirm Export')).toBeVisible()
|
||||
|
||||
const getPromiseAndResolve = () => {
|
||||
let resolve: any = () => {}
|
||||
const promise = new Promise<Download>((r) => {
|
||||
resolve = r
|
||||
})
|
||||
return [promise, resolve]
|
||||
}
|
||||
|
||||
const [downloadPromise1, downloadResolve1] = getPromiseAndResolve()
|
||||
let downloadCnt = 0
|
||||
|
||||
if (exportFrom === 'dropdown')
|
||||
page.on('download', async (download) => {
|
||||
if (downloadCnt === 0) {
|
||||
downloadResolve1(download)
|
||||
}
|
||||
downloadCnt++
|
||||
})
|
||||
await page.getByRole('button', { name: 'Submit command' }).click()
|
||||
|
||||
// This usually happens immediately after. If we're too slow we don't
|
||||
// catch it.
|
||||
await expect(page.getByText('Exported successfully')).toBeVisible()
|
||||
|
||||
if (exportFrom === 'sidebarButton' || exportFrom === 'commandBar') {
|
||||
return {
|
||||
modelPath: '',
|
||||
@ -755,15 +804,12 @@ export const doExport = async (
|
||||
}
|
||||
|
||||
// Handle download
|
||||
const download = await downloadPromise1
|
||||
const downloadLocationer = (extra = '', isImage = false) =>
|
||||
`./e2e/playwright/export-snapshots/${output.type}-${
|
||||
'storage' in output ? output.storage : ''
|
||||
}${extra}.${isImage ? 'png' : output.type}`
|
||||
const downloadLocation = downloadLocationer()
|
||||
|
||||
await download.saveAs(downloadLocation)
|
||||
|
||||
if (output.type === 'step') {
|
||||
// stable timestamps for step files
|
||||
const fileContents = await fsp.readFile(downloadLocation, 'utf-8')
|
||||
@ -772,6 +818,12 @@ export const doExport = async (
|
||||
'1970-01-01T00:00:00.0+00:00'
|
||||
)
|
||||
await fsp.writeFile(downloadLocation, newFileContents)
|
||||
} else {
|
||||
// By default all files are downloaded to the same place in playwright
|
||||
// (declared in src/lib/exportSave)
|
||||
// To remain consistent with our old web tests, we want to move some downloads
|
||||
// (images) to another directory.
|
||||
await moveDownloadedFileTo(page, downloadLocation)
|
||||
}
|
||||
|
||||
return {
|
||||
@ -798,6 +850,8 @@ export async function tearDown(page: Page, testInfo: TestInfo) {
|
||||
// It seems it's best to give the browser about 3s to close things
|
||||
// It's not super reliable but we have no real other choice for now
|
||||
await page.waitForTimeout(3000)
|
||||
|
||||
await testInfo.tronApp?.close()
|
||||
}
|
||||
|
||||
// settingsOverrides may need to be augmented to take more generic items,
|
||||
@ -808,12 +862,24 @@ export async function setup(
|
||||
testInfo?: TestInfo
|
||||
) {
|
||||
await context.addInitScript(
|
||||
async ({ token, settingsKey, settings, IS_PLAYWRIGHT_KEY }) => {
|
||||
async ({
|
||||
token,
|
||||
settingsKey,
|
||||
settings,
|
||||
IS_PLAYWRIGHT_KEY,
|
||||
PLAYWRIGHT_TEST_DIR,
|
||||
PERSIST_MODELING_CONTEXT,
|
||||
}) => {
|
||||
localStorage.clear()
|
||||
localStorage.setItem('TOKEN_PERSIST_KEY', token)
|
||||
localStorage.setItem('persistCode', ``)
|
||||
localStorage.setItem(
|
||||
PERSIST_MODELING_CONTEXT,
|
||||
JSON.stringify({ openPanes: ['code'] })
|
||||
)
|
||||
localStorage.setItem(settingsKey, settings)
|
||||
localStorage.setItem(IS_PLAYWRIGHT_KEY, 'true')
|
||||
localStorage.setItem('PLAYWRIGHT_TEST_DIR', PLAYWRIGHT_TEST_DIR)
|
||||
},
|
||||
{
|
||||
token: secrets.token,
|
||||
@ -830,6 +896,8 @@ export async function setup(
|
||||
} as Partial<SaveSettingsPayload>,
|
||||
}),
|
||||
IS_PLAYWRIGHT_KEY,
|
||||
PLAYWRIGHT_TEST_DIR: TEST_SETTINGS.app.projectDirectory,
|
||||
PERSIST_MODELING_CONTEXT,
|
||||
}
|
||||
)
|
||||
|
||||
@ -848,12 +916,15 @@ export async function setup(
|
||||
await page.emulateMedia({ reducedMotion: 'reduce' })
|
||||
|
||||
// Trigger a navigation, since loading file:// doesn't.
|
||||
await page.reload()
|
||||
// await page.reload()
|
||||
}
|
||||
|
||||
let electronApp: ElectronApplication | undefined = undefined
|
||||
let context: BrowserContext | undefined = undefined
|
||||
let page: Page | undefined = undefined
|
||||
|
||||
export async function setupElectron({
|
||||
testInfo,
|
||||
folderSetupFn,
|
||||
cleanProjectDir = true,
|
||||
appSettings,
|
||||
}: {
|
||||
@ -861,7 +932,12 @@ export async function setupElectron({
|
||||
folderSetupFn?: (projectDirName: string) => Promise<void>
|
||||
cleanProjectDir?: boolean
|
||||
appSettings?: Partial<SaveSettingsPayload>
|
||||
}) {
|
||||
}): Promise<{
|
||||
electronApp: ElectronApplication
|
||||
context: BrowserContext
|
||||
page: Page
|
||||
dir: string
|
||||
}> {
|
||||
// create or otherwise clear the folder
|
||||
const projectDirName = testInfo.outputPath('electron-test-projects-dir')
|
||||
try {
|
||||
@ -876,7 +952,7 @@ export async function setupElectron({
|
||||
await fsp.mkdir(projectDirName)
|
||||
}
|
||||
|
||||
const electronApp = await electron.launch({
|
||||
const options = {
|
||||
args: ['.', '--no-sandbox'],
|
||||
env: {
|
||||
...process.env,
|
||||
@ -886,14 +962,22 @@ export async function setupElectron({
|
||||
...(process.env.ELECTRON_OVERRIDE_DIST_PATH
|
||||
? { executablePath: process.env.ELECTRON_OVERRIDE_DIST_PATH + 'electron' }
|
||||
: {}),
|
||||
})
|
||||
const context = electronApp.context()
|
||||
const page = await electronApp.firstWindow()
|
||||
context.on('console', console.log)
|
||||
page.on('console', console.log)
|
||||
}
|
||||
|
||||
// Do this once and then reuse window on subsequent calls.
|
||||
if (!electronApp) {
|
||||
electronApp = await electron.launch(options)
|
||||
}
|
||||
|
||||
if (!context || !page) {
|
||||
context = electronApp.context()
|
||||
page = await electronApp.firstWindow()
|
||||
context.on('console', console.log)
|
||||
page.on('console', console.log)
|
||||
}
|
||||
|
||||
if (cleanProjectDir) {
|
||||
const tempSettingsFilePath = join(projectDirName, SETTINGS_FILE_NAME)
|
||||
const tempSettingsFilePath = path.join(projectDirName, SETTINGS_FILE_NAME)
|
||||
const settingsOverrides = TOML.stringify(
|
||||
appSettings
|
||||
? {
|
||||
@ -920,11 +1004,7 @@ export async function setupElectron({
|
||||
await fsp.writeFile(tempSettingsFilePath, settingsOverrides)
|
||||
}
|
||||
|
||||
await folderSetupFn?.(projectDirName)
|
||||
|
||||
await setup(context, page)
|
||||
|
||||
return { electronApp, page, dir: projectDirName }
|
||||
return { electronApp, page, context, dir: projectDirName }
|
||||
}
|
||||
|
||||
function failOnConsoleErrors(page: Page, testInfo?: TestInfo) {
|
||||
@ -1010,7 +1090,7 @@ export async function createProject({
|
||||
}
|
||||
|
||||
export function executorInputPath(fileName: string): string {
|
||||
return join('src', 'wasm-lib', 'tests', 'executor', 'inputs', fileName)
|
||||
return path.join('src', 'wasm-lib', 'tests', 'executor', 'inputs', fileName)
|
||||
}
|
||||
|
||||
export async function doAndWaitForImageDiff(
|
||||
@ -1101,3 +1181,12 @@ export function getPixelRGBs(page: Page) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
export async function pollEditorLinesSelectedLength(page: Page, lines: number) {
|
||||
return expect
|
||||
.poll(async () => {
|
||||
const lines = await page.locator('.cm-activeLine').all()
|
||||
return lines.length
|
||||
})
|
||||
.toBe(lines)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user