* initial tokeniser to wasm port
* fix tests
* add wasm to build script
* add rust tools to test action
* tweaks
* maybe tests will be happy
* tweak simple server
* trying to get tests to pass
* it pget vercel to build
* tidy up
* clean up rust files
* change lexer to use recursion so that it's consistent with the rest of the interpreter
* clean up nokeniser further
* rename variables
* readme typos
* run rust tests in CI
* follow clippy's advice
* more clippy clean up
* setup up proper serialzation to js-land
* tidy init promise in tests
* Add the ability to recast comments and some whitespace
Currently because whitespace or anything that's not needed for execution is not stored in the AST, it's hard to respect things like user formatting when recasting.
I think having a by-default-opinioned formatter is a good thing, but where this becomes problematic is when users wants to simply leave a blank space between some lines for a bit of breathing room, a code paragraph if you will, but maybe more importantly comments have not been implemented for the same reason, there wasn't a way with the current setup to insert them back in.
In some ways the most straightforward way to do this is to put whitespace and comments into the AST. Even though they are not crucial for execution, code-gen/recasting needs to be a first-class citizen in this lang so that's probably the long-term solution. However I'm trying to draw inspiration from other languages, and since it's not the norm to put comments et-al into the AST I haven't done so.
Because whitespace is tokenised already if not transformed into the AST, there is somewhat of a map of these things without going back to source code, so atm I'm experimenting with using this to insert extra linebreaks and comments back in between statements. I think this is a good compromise for the time being for what is a nice to have feature atm.
Because it's only going to respect non-code parts in between statements this will mean that you can't format objects or function params how you like (but I think this is good to have an opinioned fmt out of the box) and comments like myFunctionCall('a', /* inline comment */ b) will not work either.
* clean up
