KCL: New simulation test pipeline (#4351)
The idea behind this is to test all the various stages of executing KCL
separately, i.e.
- Start with a program
- Tokenize it
- Parse those tokens into an AST
- Recast the AST
- Execute the AST, outputting
- a PNG of the rendered model
- serialized program memory
Each of these steps reads some input and writes some output to disk.
The output of one step becomes the input to the next step. These
intermediate artifacts are also snapshotted (like expectorate or 2020)
to ensure we're aware of any changes to how KCL works. A change could
be a bug, or it could be harmless, or deliberate, but keeping it checked
into the repo means we can easily track changes.
Note: UUIDs sent back by the engine are currently nondeterministic, so
they would break all the snapshot tests. So, the snapshots use a regex
filter and replace anything that looks like a uuid with [uuid] when
writing program memory to a snapshot. In the future I hope our UUIDs will
be seedable and easy to make deterministic. At that point, we can stop
filtering the UUIDs.
We run this pipeline on many different KCL programs. Each keeps its
inputs (KCL programs), outputs (PNG, program memory snapshot) and
intermediate artifacts (AST, token lists, etc) in that directory.
I also added a new `just` command to easily generate these tests.
You can run `just new-sim-test gear $(cat gear.kcl)` to set up a new
gear test directory and generate all the intermediate artifacts for the
first time. This doesn't need any macros, it just appends some new lines
of normal Rust source code to `tests.rs`, so it's easy to see exactly
what the code is doing.
This uses `cargo insta` for convenient snapshot testing of artifacts
as JSON, and `twenty-twenty` for snapshotting PNGs.
This was heavily inspired by Predrag Gruevski's talk at EuroRust 2024
about deterministic simulation testing, and how it can both reduce bugs
and also reduce testing/CI time. Very grateful to him for chatting with
me about this over the last couple of weeks.
2024-10-30 12:14:17 -05:00
|
|
|
cnr := "cargo nextest run"
|
|
|
|
cita := "cargo insta test --accept"
|
2025-04-27 16:54:32 -07:00
|
|
|
kcl_lib_flags := "-p kcl-lib --features artifact-graph"
|
KCL: New simulation test pipeline (#4351)
The idea behind this is to test all the various stages of executing KCL
separately, i.e.
- Start with a program
- Tokenize it
- Parse those tokens into an AST
- Recast the AST
- Execute the AST, outputting
- a PNG of the rendered model
- serialized program memory
Each of these steps reads some input and writes some output to disk.
The output of one step becomes the input to the next step. These
intermediate artifacts are also snapshotted (like expectorate or 2020)
to ensure we're aware of any changes to how KCL works. A change could
be a bug, or it could be harmless, or deliberate, but keeping it checked
into the repo means we can easily track changes.
Note: UUIDs sent back by the engine are currently nondeterministic, so
they would break all the snapshot tests. So, the snapshots use a regex
filter and replace anything that looks like a uuid with [uuid] when
writing program memory to a snapshot. In the future I hope our UUIDs will
be seedable and easy to make deterministic. At that point, we can stop
filtering the UUIDs.
We run this pipeline on many different KCL programs. Each keeps its
inputs (KCL programs), outputs (PNG, program memory snapshot) and
intermediate artifacts (AST, token lists, etc) in that directory.
I also added a new `just` command to easily generate these tests.
You can run `just new-sim-test gear $(cat gear.kcl)` to set up a new
gear test directory and generate all the intermediate artifacts for the
first time. This doesn't need any macros, it just appends some new lines
of normal Rust source code to `tests.rs`, so it's easy to see exactly
what the code is doing.
This uses `cargo insta` for convenient snapshot testing of artifacts
as JSON, and `twenty-twenty` for snapshotting PNGs.
This was heavily inspired by Predrag Gruevski's talk at EuroRust 2024
about deterministic simulation testing, and how it can both reduce bugs
and also reduce testing/CI time. Very grateful to him for chatting with
me about this over the last couple of weeks.
2024-10-30 12:14:17 -05:00
|
|
|
|
2024-11-20 08:23:30 -06:00
|
|
|
# Run the same lint checks we run in CI.
|
2024-09-06 09:42:11 -05:00
|
|
|
lint:
|
2025-05-08 14:28:33 -04:00
|
|
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
2025-04-26 21:21:26 -07:00
|
|
|
# Ensure we can build without extra feature flags.
|
2025-05-08 14:28:33 -04:00
|
|
|
cargo clippy -p kcl-lib --all-targets -- -D warnings
|
2024-10-29 09:39:50 -05:00
|
|
|
|
2025-07-01 12:42:12 -05:00
|
|
|
lint-fix:
|
|
|
|
cargo clippy --workspace --all-targets --all-features --fix
|
|
|
|
|
2024-11-21 08:49:54 -06:00
|
|
|
# Run the stdlib docs generation
|
|
|
|
redo-kcl-stdlib-docs-no-imgs:
|
2025-04-26 21:21:26 -07:00
|
|
|
EXPECTORATE=overwrite {{cnr}} {{kcl_lib_flags}} docs::gen_std_tests::test_generate_stdlib
|
2024-11-21 08:49:54 -06:00
|
|
|
|
2024-11-18 14:58:33 -08:00
|
|
|
# Generate the stdlib image artifacts
|
|
|
|
# Then run the stdlib docs generation
|
2024-10-29 09:39:50 -05:00
|
|
|
redo-kcl-stdlib-docs:
|
2025-04-26 21:21:26 -07:00
|
|
|
TWENTY_TWENTY=overwrite {{cnr}} {{kcl_lib_flags}} -- kcl_test_example
|
|
|
|
TWENTY_TWENTY=overwrite {{cnr}} {{kcl_lib_flags}} docs::kcl_doc::test::kcl_test_examples
|
|
|
|
EXPECTORATE=overwrite {{cnr}} {{kcl_lib_flags}} -- docs::gen_std_tests::test_generate_stdlib
|
|
|
|
EXPECTORATE=overwrite {{cnr}} {{kcl_lib_flags}} -- generate_settings_docs
|
KCL: New simulation test pipeline (#4351)
The idea behind this is to test all the various stages of executing KCL
separately, i.e.
- Start with a program
- Tokenize it
- Parse those tokens into an AST
- Recast the AST
- Execute the AST, outputting
- a PNG of the rendered model
- serialized program memory
Each of these steps reads some input and writes some output to disk.
The output of one step becomes the input to the next step. These
intermediate artifacts are also snapshotted (like expectorate or 2020)
to ensure we're aware of any changes to how KCL works. A change could
be a bug, or it could be harmless, or deliberate, but keeping it checked
into the repo means we can easily track changes.
Note: UUIDs sent back by the engine are currently nondeterministic, so
they would break all the snapshot tests. So, the snapshots use a regex
filter and replace anything that looks like a uuid with [uuid] when
writing program memory to a snapshot. In the future I hope our UUIDs will
be seedable and easy to make deterministic. At that point, we can stop
filtering the UUIDs.
We run this pipeline on many different KCL programs. Each keeps its
inputs (KCL programs), outputs (PNG, program memory snapshot) and
intermediate artifacts (AST, token lists, etc) in that directory.
I also added a new `just` command to easily generate these tests.
You can run `just new-sim-test gear $(cat gear.kcl)` to set up a new
gear test directory and generate all the intermediate artifacts for the
first time. This doesn't need any macros, it just appends some new lines
of normal Rust source code to `tests.rs`, so it's easy to see exactly
what the code is doing.
This uses `cargo insta` for convenient snapshot testing of artifacts
as JSON, and `twenty-twenty` for snapshotting PNGs.
This was heavily inspired by Predrag Gruevski's talk at EuroRust 2024
about deterministic simulation testing, and how it can both reduce bugs
and also reduce testing/CI time. Very grateful to him for chatting with
me about this over the last couple of weeks.
2024-10-30 12:14:17 -05:00
|
|
|
|
2024-11-21 18:33:02 -06:00
|
|
|
# Copy a test KCL file from executor tests into a new simulation test.
|
|
|
|
copy-exec-test-into-sim-test test_name:
|
2025-03-01 13:59:01 -08:00
|
|
|
mkdir -p kcl-lib/tests/{{test_name}}
|
|
|
|
cp kcl-lib/e2e/executor/inputs/{{test_name}}.kcl kcl-lib/tests/{{test_name}}/input.kcl
|
|
|
|
zoo kcl fmt -w kcl-lib/tests/{{test_name}}/input.kcl
|
2024-11-21 18:33:02 -06:00
|
|
|
just new-sim-test {{test_name}}
|
|
|
|
|
2025-02-08 00:28:39 -05:00
|
|
|
# Create a new, empty KCL deterministic simulation test case.
|
2024-11-18 16:20:32 -06:00
|
|
|
new-sim-test test_name render_to_png="true":
|
2025-03-01 13:59:01 -08:00
|
|
|
mkdir kcl-lib/tests/{{test_name}}
|
|
|
|
touch kcl-lib/tests/{{test_name}}/input.kcl
|
2024-11-21 08:49:54 -06:00
|
|
|
# Add the various tests for this new test case.
|
2025-03-01 13:59:01 -08:00
|
|
|
cat kcl-lib/tests/simtest.tmpl | sed "s/TEST_NAME_HERE/{{test_name}}/" | sed "s/RENDER_TO_PNG/{{render_to_png}}/" >> kcl-lib/src/simulation_tests.rs
|
2025-02-08 00:28:39 -05:00
|
|
|
|
|
|
|
# Run a KCL deterministic simulation test case and accept output.
|
KCL: Angled line should use keyword args (#5803)
We continue migrating KCL stdlib functions to use keyword arguments. Next up is the `angledLine` family of functions (except `angledLineThatIntersects, which will be a quick follow-up).
Before vs. after:
`angledLine({angle = 90, length = 3}, %, $edge)`
=> `angledLine(angle = 90, length = 3, tag = $edge)`
`angledLineOfXLength({angle = 90, length = 3}, %, $edge)`
=> `angledLine(angle = 90, lengthX = 3, tag = $edge)`
`angledLineOfYLength({angle = 90, length = 3}, %, $edge)`
=> `angledLine(angle = 90, lengthY = 3, tag = $edge)`
`angledLineToX({angle = 90, length = 3}, %, $edge)`
=> `angledLine(angle = 90, endAbsoluteX = 3, tag = $edge)`
`angledLineToY({angle = 90, length = 3}, %, $edge)`
=> `angledLine(angle = 90, endAbsoluteY = 3, tag = $edge)`
2025-04-09 14:55:15 -05:00
|
|
|
overwrite-sim-test-sample test_name:
|
2025-05-21 10:51:31 -04:00
|
|
|
ZOO_SIM_UPDATE=always EXPECTORATE=overwrite TWENTY_TWENTY=overwrite {{cita}} {{kcl_lib_flags}} --no-quiet -- simulation_tests::kcl_samples::parse_{{test_name}}
|
|
|
|
ZOO_SIM_UPDATE=always EXPECTORATE=overwrite TWENTY_TWENTY=overwrite {{cita}} {{kcl_lib_flags}} --no-quiet -- simulation_tests::kcl_samples::unparse_{{test_name}}
|
|
|
|
ZOO_SIM_UPDATE=always EXPECTORATE=overwrite TWENTY_TWENTY=overwrite {{cita}} {{kcl_lib_flags}} --no-quiet -- simulation_tests::kcl_samples::kcl_test_execute_{{test_name}}
|
|
|
|
ZOO_SIM_UPDATE=always EXPECTORATE=overwrite TWENTY_TWENTY=overwrite {{cita}} {{kcl_lib_flags}} --no-quiet -- simulation_tests::kcl_samples::test_after_engine
|
2024-11-18 14:58:33 -08:00
|
|
|
|
2025-02-11 16:06:47 -06:00
|
|
|
overwrite-sim-test test_name:
|
2025-05-21 10:51:31 -04:00
|
|
|
ZOO_SIM_UPDATE=always EXPECTORATE=overwrite TWENTY_TWENTY=overwrite {{cita}} {{kcl_lib_flags}} --no-quiet -- simulation_tests::{{test_name}}::parse
|
|
|
|
ZOO_SIM_UPDATE=always EXPECTORATE=overwrite TWENTY_TWENTY=overwrite {{cita}} {{kcl_lib_flags}} --no-quiet -- simulation_tests::{{test_name}}::unparse
|
|
|
|
ZOO_SIM_UPDATE=always EXPECTORATE=overwrite TWENTY_TWENTY=overwrite {{cita}} {{kcl_lib_flags}} --no-quiet -- simulation_tests::{{test_name}}::kcl_test_execute
|
|
|
|
[ {{test_name}} != "kcl_samples" ] || ZOO_SIM_UPDATE=always EXPECTORATE=overwrite TWENTY_TWENTY=overwrite {{cita}} {{kcl_lib_flags}} --no-quiet -- simulation_tests::{{test_name}}::test_after_engine
|
2025-02-28 17:40:01 -08:00
|
|
|
|
2025-03-27 18:48:55 -04:00
|
|
|
# Regenerate all the simulation test output.
|
|
|
|
redo-sim-tests:
|
2025-05-21 10:51:31 -04:00
|
|
|
ZOO_SIM_UPDATE=always EXPECTORATE=overwrite TWENTY_TWENTY=overwrite {{cita}} {{kcl_lib_flags}} --no-quiet -- simulation_tests
|
2025-03-27 18:48:55 -04:00
|
|
|
|
2025-01-06 21:13:06 -05:00
|
|
|
test:
|
2025-05-07 11:35:49 -04:00
|
|
|
cargo install cargo-nextest
|
2025-05-11 00:49:55 -04:00
|
|
|
{{cnr}} --workspace --features=artifact-graph --no-fail-fast
|
2025-02-19 16:56:51 -06:00
|
|
|
|
2025-03-03 14:03:18 -08:00
|
|
|
bump-kcl-crate-versions bump='patch':
|
|
|
|
# First build the kcl-bumper tool.
|
|
|
|
cargo build -p kcl-bumper
|
|
|
|
./target/debug/kcl-bumper --bump {{bump}}
|
2025-05-02 16:55:58 -05:00
|
|
|
cargo check -p kcl-bumper # this way Cargo.lock gets updated
|
2025-03-03 14:03:18 -08:00
|
|
|
|
2025-02-19 16:56:51 -06:00
|
|
|
publish-kcl version:
|
2025-04-11 14:33:17 -07:00
|
|
|
git tag kcl-{{version}} -m "Release kcl-{{version}}"
|
2025-03-01 13:59:01 -08:00
|
|
|
cargo publish -p kcl-derive-docs
|
2025-03-10 18:04:16 -07:00
|
|
|
cargo publish -p kcl-directory-test-macro
|
2025-02-19 16:56:51 -06:00
|
|
|
cargo publish -p kcl-lib
|
|
|
|
cargo publish -p kcl-test-server
|
2025-03-01 16:38:25 -08:00
|
|
|
# We push the tag at the end of publish since pushing the tag
|
|
|
|
# will trigger CI to release the kcl python bindings.
|
|
|
|
git push origin kcl-{{version}}
|