Files
modeling-app/rust/justfile

60 lines
2.6 KiB
Makefile
Raw Normal View History

KCL: New simulation test pipeline (#4351) The idea behind this is to test all the various stages of executing KCL separately, i.e. - Start with a program - Tokenize it - Parse those tokens into an AST - Recast the AST - Execute the AST, outputting - a PNG of the rendered model - serialized program memory Each of these steps reads some input and writes some output to disk. The output of one step becomes the input to the next step. These intermediate artifacts are also snapshotted (like expectorate or 2020) to ensure we're aware of any changes to how KCL works. A change could be a bug, or it could be harmless, or deliberate, but keeping it checked into the repo means we can easily track changes. Note: UUIDs sent back by the engine are currently nondeterministic, so they would break all the snapshot tests. So, the snapshots use a regex filter and replace anything that looks like a uuid with [uuid] when writing program memory to a snapshot. In the future I hope our UUIDs will be seedable and easy to make deterministic. At that point, we can stop filtering the UUIDs. We run this pipeline on many different KCL programs. Each keeps its inputs (KCL programs), outputs (PNG, program memory snapshot) and intermediate artifacts (AST, token lists, etc) in that directory. I also added a new `just` command to easily generate these tests. You can run `just new-sim-test gear $(cat gear.kcl)` to set up a new gear test directory and generate all the intermediate artifacts for the first time. This doesn't need any macros, it just appends some new lines of normal Rust source code to `tests.rs`, so it's easy to see exactly what the code is doing. This uses `cargo insta` for convenient snapshot testing of artifacts as JSON, and `twenty-twenty` for snapshotting PNGs. This was heavily inspired by Predrag Gruevski's talk at EuroRust 2024 about deterministic simulation testing, and how it can both reduce bugs and also reduce testing/CI time. Very grateful to him for chatting with me about this over the last couple of weeks.
2024-10-30 12:14:17 -05:00
cnr := "cargo nextest run"
cita := "cargo insta test --accept"
# Run the same lint checks we run in CI.
2024-09-06 09:42:11 -05:00
lint:
cargo clippy --workspace --all-targets --tests --all-features --examples --benches -- -D warnings
# Run the stdlib docs generation
redo-kcl-stdlib-docs-no-imgs:
EXPECTORATE=overwrite {{cnr}} -p kcl-lib docs::gen_std_tests::test_generate_stdlib
# Generate the stdlib image artifacts
# Then run the stdlib docs generation
redo-kcl-stdlib-docs:
2024-12-09 18:32:00 -06:00
TWENTY_TWENTY=overwrite {{cnr}} -p kcl-lib --no-fail-fast -- kcl_test_example
EXPECTORATE=overwrite {{cnr}} -p kcl-lib --no-fail-fast -- docs::gen_std_tests::test_generate_stdlib
KCL: New simulation test pipeline (#4351) The idea behind this is to test all the various stages of executing KCL separately, i.e. - Start with a program - Tokenize it - Parse those tokens into an AST - Recast the AST - Execute the AST, outputting - a PNG of the rendered model - serialized program memory Each of these steps reads some input and writes some output to disk. The output of one step becomes the input to the next step. These intermediate artifacts are also snapshotted (like expectorate or 2020) to ensure we're aware of any changes to how KCL works. A change could be a bug, or it could be harmless, or deliberate, but keeping it checked into the repo means we can easily track changes. Note: UUIDs sent back by the engine are currently nondeterministic, so they would break all the snapshot tests. So, the snapshots use a regex filter and replace anything that looks like a uuid with [uuid] when writing program memory to a snapshot. In the future I hope our UUIDs will be seedable and easy to make deterministic. At that point, we can stop filtering the UUIDs. We run this pipeline on many different KCL programs. Each keeps its inputs (KCL programs), outputs (PNG, program memory snapshot) and intermediate artifacts (AST, token lists, etc) in that directory. I also added a new `just` command to easily generate these tests. You can run `just new-sim-test gear $(cat gear.kcl)` to set up a new gear test directory and generate all the intermediate artifacts for the first time. This doesn't need any macros, it just appends some new lines of normal Rust source code to `tests.rs`, so it's easy to see exactly what the code is doing. This uses `cargo insta` for convenient snapshot testing of artifacts as JSON, and `twenty-twenty` for snapshotting PNGs. This was heavily inspired by Predrag Gruevski's talk at EuroRust 2024 about deterministic simulation testing, and how it can both reduce bugs and also reduce testing/CI time. Very grateful to him for chatting with me about this over the last couple of weeks.
2024-10-30 12:14:17 -05:00
# Copy a test KCL file from executor tests into a new simulation test.
copy-exec-test-into-sim-test test_name:
Move the wasm lib, and cleanup rust directory and all references (#5585) * git mv src/wasm-lib rust Signed-off-by: Jess Frazelle <github@jessfraz.com> * mv wasm-lib to workspace Signed-off-by: Jess Frazelle <github@jessfraz.com> * mv kcl-lib Signed-off-by: Jess Frazelle <github@jessfraz.com> * mv derive docs Signed-off-by: Jess Frazelle <github@jessfraz.com> * resolve file paths Signed-off-by: Jess Frazelle <github@jessfraz.com> * clippy Signed-off-by: Jess Frazelle <github@jessfraz.com> * move more shit Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix more paths Signed-off-by: Jess Frazelle <github@jessfraz.com> * make yarn build:wasm work Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix scripts Signed-off-by: Jess Frazelle <github@jessfraz.com> * fixups Signed-off-by: Jess Frazelle <github@jessfraz.com> * better references Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix cargo ci Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix reference Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix more ci Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix tests Signed-off-by: Jess Frazelle <github@jessfraz.com> * cargo sort Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix script Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix Signed-off-by: Jess Frazelle <github@jessfraz.com> * fmt Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix a dep Signed-off-by: Jess Frazelle <github@jessfraz.com> * sort Signed-off-by: Jess Frazelle <github@jessfraz.com> * remove unused deps Signed-off-by: Jess Frazelle <github@jessfraz.com> * Revert "remove unused deps" This reverts commit fbabdb062e275fd5cbc1476f8480a1afee15d972. * updates Signed-off-by: Jess Frazelle <github@jessfraz.com> * deps; Signed-off-by: Jess Frazelle <github@jessfraz.com> * fixes Signed-off-by: Jess Frazelle <github@jessfraz.com> * updates Signed-off-by: Jess Frazelle <github@jessfraz.com> --------- Signed-off-by: Jess Frazelle <github@jessfraz.com>
2025-03-01 13:59:01 -08:00
mkdir -p kcl-lib/tests/{{test_name}}
cp kcl-lib/e2e/executor/inputs/{{test_name}}.kcl kcl-lib/tests/{{test_name}}/input.kcl
zoo kcl fmt -w kcl-lib/tests/{{test_name}}/input.kcl
just new-sim-test {{test_name}}
# Create a new, empty KCL deterministic simulation test case.
new-sim-test test_name render_to_png="true":
Move the wasm lib, and cleanup rust directory and all references (#5585) * git mv src/wasm-lib rust Signed-off-by: Jess Frazelle <github@jessfraz.com> * mv wasm-lib to workspace Signed-off-by: Jess Frazelle <github@jessfraz.com> * mv kcl-lib Signed-off-by: Jess Frazelle <github@jessfraz.com> * mv derive docs Signed-off-by: Jess Frazelle <github@jessfraz.com> * resolve file paths Signed-off-by: Jess Frazelle <github@jessfraz.com> * clippy Signed-off-by: Jess Frazelle <github@jessfraz.com> * move more shit Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix more paths Signed-off-by: Jess Frazelle <github@jessfraz.com> * make yarn build:wasm work Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix scripts Signed-off-by: Jess Frazelle <github@jessfraz.com> * fixups Signed-off-by: Jess Frazelle <github@jessfraz.com> * better references Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix cargo ci Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix reference Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix more ci Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix tests Signed-off-by: Jess Frazelle <github@jessfraz.com> * cargo sort Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix script Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix Signed-off-by: Jess Frazelle <github@jessfraz.com> * fmt Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix a dep Signed-off-by: Jess Frazelle <github@jessfraz.com> * sort Signed-off-by: Jess Frazelle <github@jessfraz.com> * remove unused deps Signed-off-by: Jess Frazelle <github@jessfraz.com> * Revert "remove unused deps" This reverts commit fbabdb062e275fd5cbc1476f8480a1afee15d972. * updates Signed-off-by: Jess Frazelle <github@jessfraz.com> * deps; Signed-off-by: Jess Frazelle <github@jessfraz.com> * fixes Signed-off-by: Jess Frazelle <github@jessfraz.com> * updates Signed-off-by: Jess Frazelle <github@jessfraz.com> --------- Signed-off-by: Jess Frazelle <github@jessfraz.com>
2025-03-01 13:59:01 -08:00
mkdir kcl-lib/tests/{{test_name}}
touch kcl-lib/tests/{{test_name}}/input.kcl
# Add the various tests for this new test case.
Move the wasm lib, and cleanup rust directory and all references (#5585) * git mv src/wasm-lib rust Signed-off-by: Jess Frazelle <github@jessfraz.com> * mv wasm-lib to workspace Signed-off-by: Jess Frazelle <github@jessfraz.com> * mv kcl-lib Signed-off-by: Jess Frazelle <github@jessfraz.com> * mv derive docs Signed-off-by: Jess Frazelle <github@jessfraz.com> * resolve file paths Signed-off-by: Jess Frazelle <github@jessfraz.com> * clippy Signed-off-by: Jess Frazelle <github@jessfraz.com> * move more shit Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix more paths Signed-off-by: Jess Frazelle <github@jessfraz.com> * make yarn build:wasm work Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix scripts Signed-off-by: Jess Frazelle <github@jessfraz.com> * fixups Signed-off-by: Jess Frazelle <github@jessfraz.com> * better references Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix cargo ci Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix reference Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix more ci Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix tests Signed-off-by: Jess Frazelle <github@jessfraz.com> * cargo sort Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix script Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix Signed-off-by: Jess Frazelle <github@jessfraz.com> * fmt Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix a dep Signed-off-by: Jess Frazelle <github@jessfraz.com> * sort Signed-off-by: Jess Frazelle <github@jessfraz.com> * remove unused deps Signed-off-by: Jess Frazelle <github@jessfraz.com> * Revert "remove unused deps" This reverts commit fbabdb062e275fd5cbc1476f8480a1afee15d972. * updates Signed-off-by: Jess Frazelle <github@jessfraz.com> * deps; Signed-off-by: Jess Frazelle <github@jessfraz.com> * fixes Signed-off-by: Jess Frazelle <github@jessfraz.com> * updates Signed-off-by: Jess Frazelle <github@jessfraz.com> --------- Signed-off-by: Jess Frazelle <github@jessfraz.com>
2025-03-01 13:59:01 -08:00
cat kcl-lib/tests/simtest.tmpl | sed "s/TEST_NAME_HERE/{{test_name}}/" | sed "s/RENDER_TO_PNG/{{render_to_png}}/" >> kcl-lib/src/simulation_tests.rs
# Run a KCL deterministic simulation test case and accept output.
run-sim-test test_name:
# Run all the tests for the first time, in the right order.
{{cita}} -p kcl-lib -- simulation_tests::{{test_name}}::parse
{{cita}} -p kcl-lib -- simulation_tests::{{test_name}}::unparse
KCL: New simulation test pipeline (#4351) The idea behind this is to test all the various stages of executing KCL separately, i.e. - Start with a program - Tokenize it - Parse those tokens into an AST - Recast the AST - Execute the AST, outputting - a PNG of the rendered model - serialized program memory Each of these steps reads some input and writes some output to disk. The output of one step becomes the input to the next step. These intermediate artifacts are also snapshotted (like expectorate or 2020) to ensure we're aware of any changes to how KCL works. A change could be a bug, or it could be harmless, or deliberate, but keeping it checked into the repo means we can easily track changes. Note: UUIDs sent back by the engine are currently nondeterministic, so they would break all the snapshot tests. So, the snapshots use a regex filter and replace anything that looks like a uuid with [uuid] when writing program memory to a snapshot. In the future I hope our UUIDs will be seedable and easy to make deterministic. At that point, we can stop filtering the UUIDs. We run this pipeline on many different KCL programs. Each keeps its inputs (KCL programs), outputs (PNG, program memory snapshot) and intermediate artifacts (AST, token lists, etc) in that directory. I also added a new `just` command to easily generate these tests. You can run `just new-sim-test gear $(cat gear.kcl)` to set up a new gear test directory and generate all the intermediate artifacts for the first time. This doesn't need any macros, it just appends some new lines of normal Rust source code to `tests.rs`, so it's easy to see exactly what the code is doing. This uses `cargo insta` for convenient snapshot testing of artifacts as JSON, and `twenty-twenty` for snapshotting PNGs. This was heavily inspired by Predrag Gruevski's talk at EuroRust 2024 about deterministic simulation testing, and how it can both reduce bugs and also reduce testing/CI time. Very grateful to him for chatting with me about this over the last couple of weeks.
2024-10-30 12:14:17 -05:00
TWENTY_TWENTY=overwrite {{cita}} -p kcl-lib -- tests::{{test_name}}::kcl_test_execute
overwrite-sim-test test_name:
test: Vendor kcl-samples and add simulation tests for them (#5460) * Change to unzip * Download kcl-samples as zip to public dir * Fix fetch:samples, e2e electron still not working * Change error message to be clearer * Refactor so that input and output directories of sim tests can be different * Add kcl samples test implementation * Update output since adding kcl_samples tests * Update kcl-samples branch * Fix git-ignore pattern to only apply to the root * Fix yarn install and yarn fetch:samples to work the first time * Remove unneeded exists check * Change to use kcl-samples in public directory * Add kcl-samples * Update output since updating kcl-samples * Update output files * Change to not fetch samples during yarn install * Update output after merge * Ignore kcl-samples in codespell * WIP: Don't run e2e if only kcl-samples changed * Conditionally run cargo tests * Fix to round floating point values in program memory arrays * Update output since merge and rounding numbers in memory * Fix memory redaction for floating point to find more values * Fix float redaction pattern * Update output since rounding floating point numbers * Add center to floating point pattern * Fix trigger to use picomatch syntax * Update output since rounding center * Remove kcl-samples github workflows * Enable Rust backtrace * Update output after re-running * Update output after changing order of post-extrude commands * Fix to have deterministic order of commands * Update output after reverting ordering changes * Update kcl-samples * Update output after updating samples * Fix error messages to show the names of all samples that failed * Change cargo test command to match current one * Update kcl-samples * Update output since updating kcl-samples * Add generate manifest workflow and yarn script * Fix error check to actually work * Change util function to be what we actually need * Move new files after merge * Fix paths since directory move * Add dependabot updates for kcl-samples * Add GitHub workflow to make PR to kcl-samples repo * Add GitHub workflow to check kcl-samples header comments * Fix worfklow to change to the right directory * Add auto-commit simulation test output changes * Add permissions to workflows * Fix to run git commit step * Install just if needed * Fix directory of justfile * Add installation of cargo-insta * Fix to use underscore * Fix to allow just command failure * Change to always install CLI tools and cache them * Trying to fix overwrite failing * Combine commands * Change reviewer * Change to PR targeting the next branch * Change git commands to not do unnecessary fetch * Comment out trigger for creating a PR * Update kcl-samples from next branch * Update outputs after kcl-samples change * Fix to use bash pipefail * Add rust backtrace * Print full env from sim tests * Change command to use long option name * Fix to use ci profile even when calling through just * Add INSTA_UPDATE=always * Fix git push by using an app token on checkout * Add comments * Fix to use bash options * Change to echo when no changes are found * Fix so that kcl-samples updates don't trigger full run * Fix paths to reflect new crate location * Fix path detection * Fix e2e job to ignore kcl_samples simulation test output * Fix the fetch logic for the KCL samples after vendoring (#5661) Fixes the last 2 E2E tests for #5460. --------- Co-authored-by: Pierre Jacquier <pierre@zoo.dev> Co-authored-by: Pierre Jacquier <pierrejacquier39@gmail.com> Co-authored-by: Frank Noirot <frank@zoo.dev>
2025-03-06 18:01:24 -05:00
EXPECTORATE=overwrite {{cita}} -p kcl-lib --no-quiet -- simulation_tests::{{test_name}}::parse
EXPECTORATE=overwrite {{cita}} -p kcl-lib --no-quiet -- simulation_tests::{{test_name}}::unparse
TWENTY_TWENTY=overwrite {{cita}} -p kcl-lib --no-quiet -- tests::{{test_name}}::kcl_test_execute
test:
export RUST_BRACKTRACE="full" && cargo nextest run --workspace --no-fail-fast
2025-02-19 16:56:51 -06:00
bump-kcl-crate-versions bump='patch':
# First build the kcl-bumper tool.
cargo build -p kcl-bumper
./target/debug/kcl-bumper --bump {{bump}}
2025-02-19 16:56:51 -06:00
publish-kcl version:
git tag kcl-{{version}}
Move the wasm lib, and cleanup rust directory and all references (#5585) * git mv src/wasm-lib rust Signed-off-by: Jess Frazelle <github@jessfraz.com> * mv wasm-lib to workspace Signed-off-by: Jess Frazelle <github@jessfraz.com> * mv kcl-lib Signed-off-by: Jess Frazelle <github@jessfraz.com> * mv derive docs Signed-off-by: Jess Frazelle <github@jessfraz.com> * resolve file paths Signed-off-by: Jess Frazelle <github@jessfraz.com> * clippy Signed-off-by: Jess Frazelle <github@jessfraz.com> * move more shit Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix more paths Signed-off-by: Jess Frazelle <github@jessfraz.com> * make yarn build:wasm work Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix scripts Signed-off-by: Jess Frazelle <github@jessfraz.com> * fixups Signed-off-by: Jess Frazelle <github@jessfraz.com> * better references Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix cargo ci Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix reference Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix more ci Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix tests Signed-off-by: Jess Frazelle <github@jessfraz.com> * cargo sort Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix script Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix Signed-off-by: Jess Frazelle <github@jessfraz.com> * fmt Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix a dep Signed-off-by: Jess Frazelle <github@jessfraz.com> * sort Signed-off-by: Jess Frazelle <github@jessfraz.com> * remove unused deps Signed-off-by: Jess Frazelle <github@jessfraz.com> * Revert "remove unused deps" This reverts commit fbabdb062e275fd5cbc1476f8480a1afee15d972. * updates Signed-off-by: Jess Frazelle <github@jessfraz.com> * deps; Signed-off-by: Jess Frazelle <github@jessfraz.com> * fixes Signed-off-by: Jess Frazelle <github@jessfraz.com> * updates Signed-off-by: Jess Frazelle <github@jessfraz.com> --------- Signed-off-by: Jess Frazelle <github@jessfraz.com>
2025-03-01 13:59:01 -08:00
cargo publish -p kcl-derive-docs
2025-02-19 16:56:51 -06:00
cargo publish -p kcl-lib
cargo publish -p kcl-test-server
# We push the tag at the end of publish since pushing the tag
# will trigger CI to release the kcl python bindings.
git push origin kcl-{{version}}