AI Agents Coding Review

AI is all around us, and I believe we have all tried it. I am really enjoying it, but it can be a bit hit and miss. Here are our recent experiences.

We use NetXMS, a network mapping and monitoring tool, for more complex customer networks. It provides both an API and a CLI for automating tasks, using its own proprietary scripting language, NXSL.

I like to compare AIs, so I tasked ChatGPT, Claude, GitHub Copilot, and Grok with creating some NXSL scripts. They all failed quite badly — none of them could properly work it out. It seems they couldn’t pick up the latest NXSL syntax or accurately reference the API documentation.

The same thing happened when I asked them to create an SSO adapter for single sign-on to the Ghost CMS admin portal (which runs in a subfolder, e.g. example.com/ghost). They failed to understand that if you protect the /ghost URL, you must exclude the /ghost/api subfolder — otherwise it breaks other system functionality.

In both cases, I noticed a clear inability to read and learn from the reference documentation and system guides I provided via URLs.

However, we recently ran into a different issue where one of our custom integrations broke after a core system upgrade. The upgraded version had removed the jsonpath Node.js module, which our integration depended on. I didn’t want to modify the core system itself, as that would break again with every future upgrade, and I couldn’t see an alternative solution.

The AI solved it in seconds. It identified which file was referencing the missing jsonpath module, removed the reference from the original file (left file below), and wrote a custom replacement code to simulate the missing functionality (right file below). Please see the changes highlighted in blue below.

It was an invaluable help to us.