‘I overnight switched to ChatGPT from googling’

‘I used to go to Stackoverflow.com everyday. It is weeks since I went to that site. When I posted questions, I had to wait for days before getting answers. Moreover, there are ‘high ranking’ users who keep commenting on the way I formed question than answering. Not anymore’.

There is euphoria

There is euphoria in tech industry about ChatGPT, just other rest of the segments. In domains of social sciences, politics and culture, we are aware of the ‘AI bias’ issues. But when it comes to tech, the facts can be verified. The ability to verify has caused massive shift to ChatGPT for technical answers.

At Manomaya, we keenly observed ChatGPT’s answers and performance in many technical and business domains. We observed the pattern of questions-answers in different kinds of questions – open popular technologies, advanced specialized domains, semi-proprietary topics where companies have advanced company-specific technologies improved over public technologies. And lot more.

The benefits of ChatGPT are well articulated in loads of articles and videos. The biases are also well discussed. However, the productivity improvement and the productivity loss is not really discussed well.

Here we dive deep

In past few days, we saw that in some cases, it still took several days to solve a problem. Since we use ChatGPT, it is supposed to be faster, right?

Here are some of the observations.

• When we had better domain knowledge, we got better results. When we included the right domain-specific solution-specific keywords, it got drastically better.
• We heavily use Google cloud and Azure. There are many many new updates since 2021. We get misleading answers for those. Since it is well known issue, we can live with it.
• The believable confidence: It gives very confident answers some times. Some other times, it explicitly says it is not confident. This makes the confidence believable. So, we tend to trust when it is confident. If We are not competent to verify the answers, we are likely to go down a rabbit hole. So, in a way, this is a automation tool rather than a reliable guide. That means that – it can not substitute for the lack of competencies.
• Last week a team member was arguing with me. I had an improvement suggestion for the code. He refuses to do it. His defense is clear – Because ChatGPT said it is correct. The concept of best practice is at risk. To prove and convince, it is going to take lot of time.

With google search, the members had to qualify the search results by going through the content, the domain authority of the source and so on. All of that is bypassed. Your team member has built blind faith and emotional bond with the bot. Only after trying for several days, the topic is going to come up for escalation levels.

Biggest issue is the loss of exploration habits and analysis skills.

The engineers who start in their career in ChatGPT era are at a disadvantage. Some of the tasks which required understanding and analysis, can be directly done by ChatGPT. Neither the employer, nor the customers would wait for the engineer to pick up domain knowledge and system design skills.

This poses two challenges to organizations

• Over next years, the engineers who do not get to analyze and build at grassroot level, will move up the ladder. When faced with diverse set of challenges, their weak foundation may cause issues for the entire organization.
• Internally, organizations may have solutions that are more effective. These will never be reused. The team members will keep repeating the same mistakes. The internal knowledge management will suffer.

Way Forward for Organizational Knowledge Management

Lot of companies use proprietary frameworks. These will continue to require using internal knowledge. When they hire engineers from outside who are wired to working with ChatGPT, they may struggle. The internal technologies are often documented in plain documents. If the habit of reading and writing documents is lost in talent market, it will create productivity crisis.

Even if an organization hasn’t built proprietary library, there is still proprietary code base. Particular design patterns and style may have been used. The way error handling, logging, the hierarchy of control flow and lot more may be standardized. Pasting ChatGPT output may violate lot of these. With stackoverflow.com, the engineers still need to think more before getting online solution.

Now the organizations have to start looking at building internal knowledge management bots. It need not be huge language models with billions of Machine Learning parameters.

Authentic AI research voices have stated that small model with specific focus may outperform ChatGPT or other large models. It needs to be trained with meaningful data, aligned with the purpose of the bot.

The hidden cost of using solutions with no recorded source can be quite high. It is not measured right now. Another area of productivity loss is the knowledge transfer and loss of knowledge due to employee attrition. Huge variety of issues and impacts happen due to lack of knowledge transfer.

But this cost can easily justify the ROI for specific internal bot.

Wrapping it up

It is up the organizations to start investing in this. They won’t regret this investment. The practices of knowledge transfer when there is employee churn also would get transformed with this. Domain chatbots will be an area of growth with explosive investment in coming years.