This is part of a series of blog posts that stemmed from my first experiences with “vibe coding”. For more on what I did and where you can find the rest of the posts, please refer to the first post.
Claude, and other AI Coding Assistants know SO MUCH that it’s sometimes easy to forget how much common sense they lack. They know and can quickly recite every single available java library and every function in them. However, they lack common sense and knowledge in two key areas for companies considering them:
- While they’ve read the whole public internet, they don’t know about your company. No matter how much someone knows about the outside world, when they arrive at your company they don’t know a lot of the things that you think of as second nature.
- They tend to get focused on just the problem you give them and not bring in the wider context of the application they’re building.
I experienced the first problem around deployments and testing. I use AWS CodeDeploy to deploy my code. Unfortunately, since Claude is confined to my desktop it doesn’t know that (more on this in a future blog post as it has a lot of implications beyond just what it can “know”). I also had a Jenkinsfile sitting in the code repository from before I started using CodeDeploy. Claude kept trying to update the Jenkinsfile and suggesting I do my deployments from there. This is instructive for companies thinking about how to use these tools. Don’t expect your new coding assistant to be automatically generating markdown files the way you’d like, creating unit tests, running sonar to verify metrics, etc… it doesn’t know your company’s standards unless you tell it.
The second problem with coding assistants is that they are mostly only focused on what I told them to solve. One great example of this was when I updated (as Google now requires) to using the “Sign In WIth Google” button. That worked great on my desktop site, but on the mobile version the “Sign In” text that I was using before was part of a collapsable menu bar which didn’t work at all with the button. Since I was doing the development on my desktop I didn’t notice this immediately and several users ended up complaining.
This is why I always refer to AI Assistants as extremely well read interns. A competent UI developer would not have made this mistake. They would have known how they implemented the mobile interface and would not simply have added the “Sign In With Google” button in exactly the same place that the “Sign In” text had been.
I don’t want to spoil my own thunder, but I’m going to speak about how I think this problem can be mitigated in a later post.