Files
modeling-app/rust/kcl-lib/src/test_server.rs

186 lines
6.3 KiB
Rust
Raw Normal View History

//! Types used to send data to the test server.
use std::path::PathBuf;
use crate::{
engine::new_zoo_client,
errors::ExecErrorWithState,
execution::{EnvironmentRef, ExecState, ExecutorContext, ExecutorSettings},
ConnectionError, ExecError, KclError, KclErrorWithOutputs, Program,
};
#[derive(serde::Deserialize, serde::Serialize)]
pub struct RequestBody {
pub kcl_program: String,
#[serde(default)]
pub test_name: String,
}
/// Executes a kcl program and takes a snapshot of the result.
/// This returns the bytes of the snapshot.
pub async fn execute_and_snapshot(code: &str, current_file: Option<PathBuf>) -> Result<image::DynamicImage, ExecError> {
let ctx = new_context(true, current_file).await?;
let program = Program::parse_no_errs(code).map_err(KclErrorWithOutputs::no_outputs)?;
let res = do_execute_and_snapshot(&ctx, program)
.await
.map(|(_, _, snap)| snap)
.map_err(|err| err.error);
ctx.close().await;
res
return errors back to user (#4075) * Log any Errors to stderr This isn't perfect -- in fact, this is maybe not even very good at all, but it's better than what we have today. Currently, when we get an Erorr back from the WebSocket, we drop it in kcl-lib. The web-app logs these to the console (I can't find my commit doing that off the top of my head, but I remember doing it) -- so this is some degree of partity. This won't be very useful at all for wasm usage, but it will fix issues with the zoo cli silently breaking with a "WebSocket Closed" error -- which is the same issue I was solving for in the desktop app too. In the future perhaps this can be a real Error? I'm not totally sure yet, since we can't align to the request-id, so we can't really tie it to a specific call (yet). * add to responses Signed-off-by: Jess Frazelle <github@jessfraz.com> * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest) * add a test Signed-off-by: Jess Frazelle <github@jessfraz.com> * clippy[ Signed-off-by: Jess Frazelle <github@jessfraz.com> * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest) * empty * fix error Signed-off-by: Jess Frazelle <github@jessfraz.com> * updates tests Signed-off-by: Jess Frazelle <github@jessfraz.com> * docs Signed-off-by: Jess Frazelle <github@jessfraz.com> --------- Signed-off-by: Jess Frazelle <github@jessfraz.com> Co-authored-by: Jess Frazelle <github@jessfraz.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jess Frazelle <jessfraz@users.noreply.github.com>
2024-10-03 01:05:12 -04:00
}
KCL: New simulation test pipeline (#4351) The idea behind this is to test all the various stages of executing KCL separately, i.e. - Start with a program - Tokenize it - Parse those tokens into an AST - Recast the AST - Execute the AST, outputting - a PNG of the rendered model - serialized program memory Each of these steps reads some input and writes some output to disk. The output of one step becomes the input to the next step. These intermediate artifacts are also snapshotted (like expectorate or 2020) to ensure we're aware of any changes to how KCL works. A change could be a bug, or it could be harmless, or deliberate, but keeping it checked into the repo means we can easily track changes. Note: UUIDs sent back by the engine are currently nondeterministic, so they would break all the snapshot tests. So, the snapshots use a regex filter and replace anything that looks like a uuid with [uuid] when writing program memory to a snapshot. In the future I hope our UUIDs will be seedable and easy to make deterministic. At that point, we can stop filtering the UUIDs. We run this pipeline on many different KCL programs. Each keeps its inputs (KCL programs), outputs (PNG, program memory snapshot) and intermediate artifacts (AST, token lists, etc) in that directory. I also added a new `just` command to easily generate these tests. You can run `just new-sim-test gear $(cat gear.kcl)` to set up a new gear test directory and generate all the intermediate artifacts for the first time. This doesn't need any macros, it just appends some new lines of normal Rust source code to `tests.rs`, so it's easy to see exactly what the code is doing. This uses `cargo insta` for convenient snapshot testing of artifacts as JSON, and `twenty-twenty` for snapshotting PNGs. This was heavily inspired by Predrag Gruevski's talk at EuroRust 2024 about deterministic simulation testing, and how it can both reduce bugs and also reduce testing/CI time. Very grateful to him for chatting with me about this over the last couple of weeks.
2024-10-30 12:14:17 -05:00
/// Executes a kcl program and takes a snapshot of the result.
/// This returns the bytes of the snapshot.
pub async fn execute_and_snapshot_ast(
ast: Program,
current_file: Option<PathBuf>,
with_export_step: bool,
) -> Result<(ExecState, EnvironmentRef, image::DynamicImage, Option<Vec<u8>>), ExecErrorWithState> {
let ctx = new_context(true, current_file).await?;
let (exec_state, env, img) = match do_execute_and_snapshot(&ctx, ast).await {
Ok((exec_state, env_ref, img)) => (exec_state, env_ref, img),
Err(err) => {
// If there was an error executing the program, return it.
// Close the context to avoid any resource leaks.
ctx.close().await;
return Err(err);
}
};
let mut step = None;
if with_export_step {
let files = match ctx.export_step(true).await {
Ok(f) => f,
Err(err) => {
// Close the context to avoid any resource leaks.
ctx.close().await;
return Err(ExecErrorWithState::new(
ExecError::BadExport(format!("Export failed: {:?}", err)),
exec_state.clone(),
));
}
};
step = files.into_iter().next().map(|f| f.contents);
}
ctx.close().await;
Ok((exec_state, env, img, step))
return errors back to user (#4075) * Log any Errors to stderr This isn't perfect -- in fact, this is maybe not even very good at all, but it's better than what we have today. Currently, when we get an Erorr back from the WebSocket, we drop it in kcl-lib. The web-app logs these to the console (I can't find my commit doing that off the top of my head, but I remember doing it) -- so this is some degree of partity. This won't be very useful at all for wasm usage, but it will fix issues with the zoo cli silently breaking with a "WebSocket Closed" error -- which is the same issue I was solving for in the desktop app too. In the future perhaps this can be a real Error? I'm not totally sure yet, since we can't align to the request-id, so we can't really tie it to a specific call (yet). * add to responses Signed-off-by: Jess Frazelle <github@jessfraz.com> * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest) * add a test Signed-off-by: Jess Frazelle <github@jessfraz.com> * clippy[ Signed-off-by: Jess Frazelle <github@jessfraz.com> * A snapshot a day keeps the bugs away! 📷🐛 (OS: ubuntu-latest) * empty * fix error Signed-off-by: Jess Frazelle <github@jessfraz.com> * updates tests Signed-off-by: Jess Frazelle <github@jessfraz.com> * docs Signed-off-by: Jess Frazelle <github@jessfraz.com> --------- Signed-off-by: Jess Frazelle <github@jessfraz.com> Co-authored-by: Jess Frazelle <github@jessfraz.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jess Frazelle <jessfraz@users.noreply.github.com>
2024-10-03 01:05:12 -04:00
}
pub async fn execute_and_snapshot_no_auth(
code: &str,
current_file: Option<PathBuf>,
) -> Result<(image::DynamicImage, EnvironmentRef), ExecError> {
let ctx = new_context(false, current_file).await?;
let program = Program::parse_no_errs(code).map_err(KclErrorWithOutputs::no_outputs)?;
let res = do_execute_and_snapshot(&ctx, program)
.await
.map(|(_, env_ref, snap)| (snap, env_ref))
.map_err(|err| err.error);
ctx.close().await;
res
KCL: New simulation test pipeline (#4351) The idea behind this is to test all the various stages of executing KCL separately, i.e. - Start with a program - Tokenize it - Parse those tokens into an AST - Recast the AST - Execute the AST, outputting - a PNG of the rendered model - serialized program memory Each of these steps reads some input and writes some output to disk. The output of one step becomes the input to the next step. These intermediate artifacts are also snapshotted (like expectorate or 2020) to ensure we're aware of any changes to how KCL works. A change could be a bug, or it could be harmless, or deliberate, but keeping it checked into the repo means we can easily track changes. Note: UUIDs sent back by the engine are currently nondeterministic, so they would break all the snapshot tests. So, the snapshots use a regex filter and replace anything that looks like a uuid with [uuid] when writing program memory to a snapshot. In the future I hope our UUIDs will be seedable and easy to make deterministic. At that point, we can stop filtering the UUIDs. We run this pipeline on many different KCL programs. Each keeps its inputs (KCL programs), outputs (PNG, program memory snapshot) and intermediate artifacts (AST, token lists, etc) in that directory. I also added a new `just` command to easily generate these tests. You can run `just new-sim-test gear $(cat gear.kcl)` to set up a new gear test directory and generate all the intermediate artifacts for the first time. This doesn't need any macros, it just appends some new lines of normal Rust source code to `tests.rs`, so it's easy to see exactly what the code is doing. This uses `cargo insta` for convenient snapshot testing of artifacts as JSON, and `twenty-twenty` for snapshotting PNGs. This was heavily inspired by Predrag Gruevski's talk at EuroRust 2024 about deterministic simulation testing, and how it can both reduce bugs and also reduce testing/CI time. Very grateful to him for chatting with me about this over the last couple of weeks.
2024-10-30 12:14:17 -05:00
}
KCL: New simulation test pipeline (#4351) The idea behind this is to test all the various stages of executing KCL separately, i.e. - Start with a program - Tokenize it - Parse those tokens into an AST - Recast the AST - Execute the AST, outputting - a PNG of the rendered model - serialized program memory Each of these steps reads some input and writes some output to disk. The output of one step becomes the input to the next step. These intermediate artifacts are also snapshotted (like expectorate or 2020) to ensure we're aware of any changes to how KCL works. A change could be a bug, or it could be harmless, or deliberate, but keeping it checked into the repo means we can easily track changes. Note: UUIDs sent back by the engine are currently nondeterministic, so they would break all the snapshot tests. So, the snapshots use a regex filter and replace anything that looks like a uuid with [uuid] when writing program memory to a snapshot. In the future I hope our UUIDs will be seedable and easy to make deterministic. At that point, we can stop filtering the UUIDs. We run this pipeline on many different KCL programs. Each keeps its inputs (KCL programs), outputs (PNG, program memory snapshot) and intermediate artifacts (AST, token lists, etc) in that directory. I also added a new `just` command to easily generate these tests. You can run `just new-sim-test gear $(cat gear.kcl)` to set up a new gear test directory and generate all the intermediate artifacts for the first time. This doesn't need any macros, it just appends some new lines of normal Rust source code to `tests.rs`, so it's easy to see exactly what the code is doing. This uses `cargo insta` for convenient snapshot testing of artifacts as JSON, and `twenty-twenty` for snapshotting PNGs. This was heavily inspired by Predrag Gruevski's talk at EuroRust 2024 about deterministic simulation testing, and how it can both reduce bugs and also reduce testing/CI time. Very grateful to him for chatting with me about this over the last couple of weeks.
2024-10-30 12:14:17 -05:00
async fn do_execute_and_snapshot(
ctx: &ExecutorContext,
program: Program,
) -> Result<(ExecState, EnvironmentRef, image::DynamicImage), ExecErrorWithState> {
let mut exec_state = ExecState::new(ctx);
let result = ctx
parallelized modules from kcl (#6206) * wip * sketch a bit more; going to pull this out of tests next * wip * lock start things * this was a bad idea * Revert "this was a bad idea" This reverts commit a2092e7ed664f2be17eb553e82d39a16983f52ed. * prepare prelude before spawning * error * poop * yike * :( * ok * Reapply "this was a bad idea" This reverts commit fafdf410935abbddd41f312474322a79222a523f. * chip away more * man this is bad * fix rebase add feature flag Signed-off-by: Jess Frazelle <github@jessfraz.com> * get rid of execution kind Signed-off-by: Jess Frazelle <github@jessfraz.com> * clippy Signed-off-by: Jess Frazelle <github@jessfraz.com> * logs Signed-off-by: Jess Frazelle <github@jessfraz.com> * updates Signed-off-by: Jess Frazelle <github@jessfraz.com> * no extra executes Signed-off-by: Jess Frazelle <github@jessfraz.com> * race w batch Signed-off-by: Jess Frazelle <github@jessfraz.com> * cluppy Signed-off-by: Jess Frazelle <github@jessfraz.com> * no printlns Signed-off-by: Jess Frazelle <github@jessfraz.com> * no printlns Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix source ranges Signed-off-by: Jess Frazelle <github@jessfraz.com> * batch shit Signed-off-by: Jess Frazelle <github@jessfraz.com> * fixes Signed-off-by: Jess Frazelle <github@jessfraz.com> * updates Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix some bugs Signed-off-by: Jess Frazelle <github@jessfraz.com> * fix error Signed-off-by: Jess Frazelle <github@jessfraz.com> * cut 1 * preserve mem * re-ad deep_clone the helper we were calling was pushing a new call, which was hanging out. we can skip the middleman since we already have something properly prepared, just without a stdlib in some cases. * skip non-kcl * clean up source range bug * error message changed the uuids also changed because the error is hit before execute even starts. * typo * rensnapshot a few * order things * MAYBE REVERT LATER: attempt at an ordering * snapsnap * Revert "snapsnap" This reverts commit 7350b32c7d0c90c1dd0352aa4a5686370ea422fd. * Revert "MAYBE REVERT LATER:" This reverts commit ab49f3e85ff5c9dd2c625eb11573f82877239104. * ugh * poop * poop2 * lint * tranche 1 * more * more snaps * snap * more * update * MAYBE REVERT THIS * cache multi-file Signed-off-by: Jess Frazelle <github@jessfraz.com> * addd tests Signed-off-by: Jess Frazelle <github@jessfraz.com> * set to false Signed-off-by: Jess Frazelle <github@jessfraz.com> * add test outputs Signed-off-by: Jess Frazelle <github@jessfraz.com> * clippy Signed-off-by: Jess Frazelle <github@jessfraz.com> * kcl-py-bindings uses carwheel Signed-off-by: Jess Frazelle <github@jessfraz.com> * update snapshots Signed-off-by: Jess Frazelle <github@jessfraz.com> * updates Signed-off-by: Jess Frazelle <github@jessfraz.com> --------- Signed-off-by: Jess Frazelle <github@jessfraz.com> Co-authored-by: Paul R. Tagliamonte <paul@zoo.dev> Co-authored-by: Paul Tagliamonte <paultag@gmail.com>
2025-04-16 11:52:14 -07:00
.run_single_threaded(&program, &mut exec_state)
.await
.map_err(|err| ExecErrorWithState::new(err.into(), exec_state.clone()))?;
for e in exec_state.errors() {
if e.severity.is_err() {
return Err(ExecErrorWithState::new(
KclErrorWithOutputs::no_outputs(KclError::Semantic(e.clone().into())).into(),
exec_state.clone(),
));
}
}
let snapshot_png_bytes = ctx
.prepare_snapshot()
.await
.map_err(|err| ExecErrorWithState::new(err, exec_state.clone()))?
.contents
.0;
// Decode the snapshot, return it.
let img = image::ImageReader::new(std::io::Cursor::new(snapshot_png_bytes))
.with_guessed_format()
.map_err(|e| ExecError::BadPng(e.to_string()))
.and_then(|x| x.decode().map_err(|e| ExecError::BadPng(e.to_string())))
.map_err(|err| ExecErrorWithState::new(err, exec_state.clone()))?;
Ok((exec_state, result.0, img))
}
pub async fn new_context(with_auth: bool, current_file: Option<PathBuf>) -> Result<ExecutorContext, ConnectionError> {
let mut client = new_zoo_client(if with_auth { None } else { Some("bad_token".to_string()) }, None)
.map_err(ConnectionError::CouldNotMakeClient)?;
if !with_auth {
// Use prod, don't override based on env vars.
// We do this so even in the engine repo, tests that need to run with
// no auth can fail in the same way as they would in prod.
client.set_base_url("https://api.zoo.dev".to_string());
}
let mut settings = ExecutorSettings {
highlight_edges: true,
enable_ssao: false,
show_grid: false,
replay: None,
project_directory: None,
current_file: None,
};
if let Some(current_file) = current_file {
settings.with_current_file(current_file);
}
let ctx = ExecutorContext::new(&client, settings)
.await
.map_err(ConnectionError::Establishing)?;
Ok(ctx)
}
pub async fn execute_and_export_step(
code: &str,
current_file: Option<PathBuf>,
) -> Result<
(
ExecState,
EnvironmentRef,
Vec<kittycad_modeling_cmds::websocket::RawFile>,
),
ExecErrorWithState,
> {
let ctx = new_context(true, current_file).await?;
let mut exec_state = ExecState::new(&ctx);
let program = Program::parse_no_errs(code)
.map_err(|err| ExecErrorWithState::new(KclErrorWithOutputs::no_outputs(err).into(), exec_state.clone()))?;
let result = ctx
.run(&program, &mut exec_state)
.await
.map_err(|err| ExecErrorWithState::new(err.into(), exec_state.clone()))?;
for e in exec_state.errors() {
if e.severity.is_err() {
return Err(ExecErrorWithState::new(
KclErrorWithOutputs::no_outputs(KclError::Semantic(e.clone().into())).into(),
exec_state.clone(),
));
}
}
let files = match ctx.export_step(true).await {
Ok(f) => f,
Err(err) => {
return Err(ExecErrorWithState::new(
ExecError::BadExport(format!("Export failed: {:?}", err)),
exec_state.clone(),
));
}
};
ctx.close().await;
Ok((exec_state, result.0, files))
}