Conjure JavaScript Support

[Stansom]

Hi, I've added JavaScript support for Conjure. Because I use JS quite a lot, REPL support really helps me move faster and make fewer mistakes, providing a pretty similar experience to Clojure.

However, the Node.js REPL doesn't allow "dynamic rebinding" of vars. Functions declared using the "const fnName = (...args) =>" form are not re-declarable. I've introduced some transformations from the "const" form into the standard JS function form, so you don't have to constantly keep that Node.js quirk in mind. Also, the Node.js REPL doesn't support "import". My solution is to change "import" into the "const importName = require(...)" form, so you can use it easily.

JavaScript support helps me make quick sketches and experiments, and test my code immediately, just like I do in Clojure. I hope someone else finds it helpful too.

[russtoku]

A quick peek looks good so far.

Would you be able to add a dev/javascript/sandbox.js file?

It should contain some small examples of JavaScript code that can evaluated successfully in the client. The examples should give a sense of the expected behavior as well as help future contributors. Maybe show code that can't be evaluated so that a contributor in the future could resolve the problem.

[Stansom]

Hi! Sandbox has been added.

[russtoku]

The sandbox.js file looks great! Even with async stuff.

Almost all of the lines in sandbox.js can be evaluated with <localleader>ee except:

• import statements
• class definition

The import statements and class definition have to be selected and evaluated with <localleader>E.

Thanks for creating a very nice client! Works smoothly from what I can tell (I'm an infrequent user of JavaScript).

[russtoku]

I have some suggestions:

• To make it so that <localleader>ee works with import statements and class definintions, add the following line to the form-node? function:

  (= :import_statement (node:type)) (= :class_declaration (node:type))

  It can be added after the line that reads:

  (= :try_statement (node:type)) (= :expression_statement (node:type))

• Format the Fennel code like the other clients. This is for consistency. While there is a Fennel style guide, it may be easier to follow the other clients' formatting.
• Try to handle messages sent out via stderr. This code will send a string out via stderr but nothing appears in the log.

  process.stderr.write("error! some error occurred\n");

• Use nfnl's define to make the client reloadable. This makes it easier to develop and maintain the client as you won't have to restart Neovim to see the effects of changes to the Fennel code. For an example, see fnl/conjure/client/guile/socket.fnl.

[Stansom]

• I usually use <localleader>er and didn't even notice that import and class don't work with <localleader>ee. Fixed.
• Formatted the code to match other clients style.
• Used nfnl's define, but I still don't understand how it works. 😅
• Regarding processing messages in stderr, unfortunately, I couldn't figure out where I could intercept them to output to the REPL.

[russtoku]

@Olical has a guide for Reloadable Fennel in Neovim.

I hope that I got this correct. I believe the way it works is the definemacro in nfnl sets things up so that using a #(M.func) form will call the M.func in the most recently compiled version of the M module rather than the version of the module that existed when Neovim was started up for the session. In other words, a bit of indirection magic calls the current compiled version.

I found that you first compile the module's file or the entire buffer (of the module's file if you're editing it). After that, you can switch back to a code buffer and see the new behavior of the client when you evaluate pieces of code.

I find that another benefit of using nfnl's define macro is that it's easier to know which functions and variables are publicly visible when you're reading or working with the code. Otherwise, you'd have to repeatedly jump to the bottom of the file to see which ones are included in the table that's returned when the Lua file is required.

[russtoku]

Test your latest changes and they work.

I think you don't have to worry too much about processing messages via stderr. That can come later. What you have now works well.

I'll dig into my old notes and see if I can come up with some suggestions.

[Stansom]

@russtoku Thank you for your interest and suggestions! I will try to tinker with stderr later.
Found a solution for displaying stderr to the REPL. Maybe there's a better way, I don't know. 😁

[russtoku]

I just tested your recent changes and they work nicely!

[Olical]

A great start! I have some comments on some Fennel code but I'll help you work through that where required. Thanks for putting the effort into learning so much for this! I know it's a bit of a mountain to climb but you're doing great so far.

[Olical]

And thanks for jumping in here to help while I was busy all week @russtoku! Much appreciated as always!

[Olical]

Giving this a go locally and it seems to be working great for the most part. I ran into an issue with <prefix>ef on the sandbox file though, I get the following stack trace.

E5108: Error executing lua: Vim:Error executing Lua callback: ...s/Olical/conjure/lua/conjure/client/javascript/stdio.lua:110: attempt to concatenate local 'decl' (a nil value)                                                      
stack traceback:                                                                                                                                                                                                                                
        ...s/Olical/conjure/lua/conjure/client/javascript/stdio.lua:110: in function 'replace_arrows'                                                                                                                                           
        ...s/Olical/conjure/lua/conjure/client/javascript/stdio.lua:138: in function 'prep_code'                                                                                                                                                
        ...s/Olical/conjure/lua/conjure/client/javascript/stdio.lua:188: in function <...s/Olical/conjure/lua/conjure/client/javascript/stdio.lua:178>                                                                                          
        [C]: at 0x5eb7df020c30                                                                                                                                                                                                                  
stack traceback:                                                                                                                                                                                                                                
        [C]: at 0x5eb7df020c30

Looks like this if statement just falls through to a nil sometimes then when we try to concat it with a string it throws an exception because (.. nil "foo") throws. Ways around this include changing the if statement so a string is always in decl or maybe using conjure.nfnl.core which has a join function that works a bit like .. but is safe with all kinds of values like nil and can take an optional separator to put between each value.

[Olical]

If I delete the very first line from the sandbox and re-eval the whole file I get the following:

// eval (file): /home/olical/repos/Olical/conjure/dev/javascript/sandbox.js
(out) | | | | | | | | | let a = "foo";
(out)     ^
(out) Uncaught SyntaxError: Identifier 'a' has already been declared

Which is interesting!

[Stansom]

@Olical Hi! Yes it was definitely a bug that I fixed, now <localleader>ef works.

// eval (file): /home/olical/repos/Olical/conjure/dev/javascript/sandbox.js
(out) | | | | | | | | | let a = "foo";
(out)     ^
(out) Uncaught SyntaxError: Identifier 'a' has already been declared

And that one is a result of the first one.

[russtoku]

Some small suggestions.

• For people who read the help doc, it might be a good thing to mention the "hacks" that are performed on the JavaScript code when the code is evaluated. It would make the behavior less surprising because the user's code is changed behind their back to work around limitations in the Node.js REPL.

  I would say something like this at the end of the Introduction section:

  This client automatically changes imports and arrow functions to work around some of the limitations of the Node.js REPL.

• Although stated in the warning-msg function, it might be helpful for future maintainers to add a comment to the replace-imports and replace-arrows functions to explain that these functions are there to make interaction with the Node.js REPL better.

  I would say something like:

  ;; Replace imports for the user so that the Node.js REPL doesn't complain.
  ;; See https://github.com/nodejs/node/issues/48084
  fn replace-imports [s]
     ...

  ;; Rewrite arrow functions for the user so that the functions can be redefined in Node.js REPL.
  fn replace-arrows [s]
     ...

[russtoku]

I ran through the sandbox.js file and things work as expected.

[Stansom]

Hello @russtoku, your suggestions are always on point! Thank you.

[gbroques]

I just found out about Conjure a day ago, and got it setup to help write and evaluate Lua code for Neovim.

It's amazing!

I also write a fair amount of JavaScript, and I'm looking forward to these changes. Thank you @Stansom for initiating it.

It's great to see all the collaboration and support in this PR from @russtoku and @Olical.

Great work guys. 👏

[russtoku]

@gbroques, Thanks for the feedback!

[russtoku]

@Stansom, I ran through the sandbox.js agin and things are still working.

[russtoku]

You have a duplicate key mapping for <localleader>cs. In the config, it says:

(config.merge
  {:client
    {:javascript
      {:stdio
        {:mapping {:start :cs
                   :stop :cS
                   :restart :cr
                   :interrupt :ei
                   :stray :cs}}}}}))

so both :start and :stray are mapped to <localleader>cs. This causes only the last one to survive so you can only toggle stray output and not start the REPL after stopping it.

The Neovim command, :nmap <localleader> will show that <localleader>cs only toggles the stray output.

When I changed the key mapping for :stray, toggling the stray output does work.

Might I suggest using :ts for the key mapping to toggle stray output?

[Stansom]

@russtoku Ohhhh, I'm blind... Thank you for catching the bug.

[Olical]

Code and docs look good, it works great for me, this is a massive addition. Excellent work everyone! I'm really happy with this.

I want to find a way to remove the out prefix and mark things as evaluation results in some cases, but that requires some better stdio wrapping code. That's something I'll provide at some point and we can migrate clients to it. You've done a great job with the stdio wrapping code available right now, hopefully I can provide something more robust and smart some day soon!

[Olical]

Should be ts but I'm updating that. Great addition for this language specifically. This is why we have clients with lots of freedom, mappings that make sense for one might not for others, we have wiggle room to get specific for each language and community 🎉

[Stansom]

@Olical I wanted to thank you for this fantastic plugin. It was the main reason I switched from VSCode to NeoVim. I really wanted that quick feedback loop when working with Clojure, something VSCode's equivalent plugins couldn't provide at the time. When I saw how simple it was to work with the Clojure REPL in NeoVim using Conjure, I was instantly sold. It took some time to get used to, but the benefits have been huge.

However, when working with JavaScript, I still had to go back to VSCode to get anything resembling “REPL-like” features. I wanted to write JavaScript in the “Clojure way,” with the same tight feedback loop for exploring and experimenting with code. Unfortunately, VSCode plugins couldn’t match the experience, they lacked proper support for imports and top-level asynchronous code.

One day, I decided to look into Conjure’s source code to see if I could implement a client for a JavaScript REPL. I was delighted to find the sources were easy to understand. Your plugin's architecture is incredibly extensible, and the surrounding ecosystem is so well-designed that it allowed me to move quickly and have a great time building the client. The project is well-structured, the automatic compilation from Fennel to Lua is beautiful, and writing Fennel with Conjure is just so cool and fun. Thank you for creating such a great tool!

Thank you as well, @russtoku . I really appreciate your quick feedback and helpful suggestions. Your involvement made a big difference!

I hope this small contribution helps JavaScript developers write code faster and enjoy the “LISP way” of coding, with instant feedback and plenty of joy.