Add the ability to recast comments and some whitespace (#10)

* Add the ability to recast comments and some whitespace

Currently because whitespace or anything that's not needed for execution is not stored in the AST, it's hard to respect things like user formatting when recasting.

I think having a by-default-opinioned formatter is a good thing, but where this becomes problematic is when users wants to simply leave a blank space between some lines for a bit of breathing room, a code paragraph if you will, but maybe more importantly comments have not been implemented for the same reason, there wasn't a way with the current setup to insert them back in.

In some ways the most straightforward way to do this is to put whitespace and comments into the AST. Even though they are not crucial for execution, code-gen/recasting needs to be a first-class citizen in this lang so that's probably the long-term solution. However I'm trying to draw inspiration from other languages, and since it's not the norm to put comments et-al into the AST I haven't done so.

Because whitespace is tokenised already if not transformed into the AST, there is somewhat of a map of these things without going back to source code, so atm I'm experimenting with using this to insert extra linebreaks and comments back in between statements. I think this is a good compromise for the time being for what is a nice to have feature atm.

Because it's only going to respect non-code parts in between statements this will mean that you can't format objects or function params how you like (but I think this is good to have an opinioned fmt out of the box) and comments like myFunctionCall('a', /* inline comment */ b) will not work either.

* clean up

src/lang/abstractSyntaxTree.test.ts

@@ -1853,3 +1853,91 @@ describe('testing findEndofBinaryExpression', () => {
     expect(end).toBe(code.indexOf('))') + 1)
   })
 })
+
+describe('testing code with comments', () => {
+  it('should ignore line comments', () => {
+    const comment = '// this is a comment'
+    const codeWithComment = `const yo = 5
+${comment}
+const yo2 = 6`
+    // filling with extra whitespace to make the source start end numbers match
+    const codeWithoutComment = `const yo = 5
+${comment
+  .split('')
+  .map(() => ' ')
+  .join('')}
+const yo2 = 6`
+    const { body } = abstractSyntaxTree(lexer(codeWithComment))
+    const { body: bodyWithoutComment } = abstractSyntaxTree(
+      lexer(codeWithoutComment)
+    )
+    expect(body).toEqual(bodyWithoutComment)
+  })
+  it('should ignore block comments', () => {
+    const comment = `/* this is a
+multi line
+comment */`
+    const codeWithComment = `const yo = 5${comment}
+const yo2 = 6`
+    // filling with extra whitespace to make the source start end numbers match
+    const codeWithoutComment = `const yo = 5${comment
+      .split('')
+      .map(() => ' ')
+      .join('')}
+const yo2 = 6`
+    const { body } = abstractSyntaxTree(lexer(codeWithComment))
+    const { body: bodyWithoutComment } = abstractSyntaxTree(
+      lexer(codeWithoutComment)
+    )
+    expect(body).toEqual(bodyWithoutComment)
+  })
+  it('comment in function declaration', () => {
+    const code = `const yo=(a)=>{
+  // this is a comment
+  return a
+}`
+    const { body } = abstractSyntaxTree(lexer(code))
+    const yo = [
+      {
+        type: 'VariableDeclaration',
+        start: 0,
+        end: 51,
+        kind: 'const',
+        declarations: [
+          {
+            type: 'VariableDeclarator',
+            start: 6,
+            end: 51,
+            id: { type: 'Identifier', start: 6, end: 8, name: 'yo' },
+            init: {
+              type: 'FunctionExpression',
+              start: 9,
+              end: 51,
+              id: null,
+              params: [{ type: 'Identifier', start: 10, end: 11, name: 'a' }],
+              body: {
+                type: 'BlockStatement',
+                start: 14,
+                end: 51,
+                body: [
+                  {
+                    type: 'ReturnStatement',
+                    start: 41,
+                    end: 49,
+                    argument: {
+                      type: 'Identifier',
+                      start: 48,
+                      end: 49,
+                      name: 'a',
+                    },
+                  },
+                ],
+              },
+            },
+          },
+        ],
+      },
+    ]
+    expect(body).toEqual(yo)
+  })
+})