It is hard to keep up with the number of companies announcing releases of software integrated with ChatGPT. Last week we asked if we are ready for ChatGPT so this week we emphasise the structured approach that must be taken.

Creating meaningful results with ChatGPT isn’t difficult, but it must be structured.

By: Christopher Surdak, JD

In part two of this discussion about the new technology phenomenon known as ChatGPT, the author explores five key practices that organizations should apply to get the most out of using this tool. These practices are based on over two decades of work in artificial intelligence, natural language processing, and unstructured data analytics, and will hopefully assist users in extracting productivity out of this technology, rather than mere entertainment.

1. Begin With the End In Mind

Anyone who has been exploring what is possible with ChatGPT has likely come to one definitive conclusion: it can be a tremendous time sink. Interactions with the system can be enlightening, engaging and maybe even a bit creepy. But, overall one can spend an inordinate amount of time interacting with the technology without necessarily creating productive outcomes.

To prevent such productivity loss, it is important to have specific goals when engaging with ChatGPT. While the process of exploration may be fun, there needs to be an end in mind. This is particularly true if your organization is seeking to use the technology across the entire business. Many of us are familiar with the analysis-paralysis that can overwhelm an organization’s processes. Such loss of productivity is frequently the result of large-scale “data democratization” efforts, as more and more people spend more and more time analyzing and less time actually doing. There is a significant danger of such productivity losses with ChatGPT, given its conversational interface and social interaction. So, when using ChatGPT try to execute with it, rather than explore with it.

ChatGPT represents a capability to get the first eighty percent of content-based work completed nearly instantly. The question you must ask and answer is this: Are you going to use this tool to increase operational speed and reduce costs, or are you going to use it to improve operational results? If the goal is to increase speed and lower costs, this tool’s eighty percent answer must be accepted as “good enough,” and rapidly used. If instead, the goal is to improve quality, then the speed of ChatGPT may cause significant issues, as the ability to generate partial results rapidly will put even greater pressure on your most talented humans to contribute their additional twenty percent of high-quality material, even as the queue from their digital coworker grows deeper and deeper.

Trust But Verify

President Reagan’s famous saying on nuclear disarmament, “trust but verify,” certainly applies with using popularity engines like ChatGPT. Since this system presents a combination of the most-average and most-popular responses to your queries you aren’t necessarily getting the best answers. ChatGPT can improve a below average response, but rarely will it provide an exceptional response. This can prove very heuilpful to a person who is a neophyte on the question at hand, but will not likely generate better results for an expert on the same topic.

There is a chance that ChatGPT will continue to improve over time, but only if those better responses happen to also be the most popular. Hence, it is important to not take ChatGPT results at their face value. Any output from the system that you intend to rely upon should be independently verified through some other source. Indeed, any discrepancies that you may find should be back filtered through ChatGPT both to see if it can maintain some referential integrity (e.g., understand and contextualize the discrepancy), and to provide the feedback that ChatGPT is designed to leverage to improve its results.  

Consistent, if Not Common

When assessing responses from ChatGPT it must be remembered that not all responses are created equally. A consensus amongst a small number of responses is typically more reliable than variation amongst a large number of responses. This is the experts-versus-wisdom-of-the-crowd issue, and ChatGPT brings this contrast to the fore.

Asking ChatGPT for such details behind its responses can give clarity to the quality of its results, and this should be standard practice as you interact with the system. Your queries should ask where ChatGPT sourced its responses, from how many citations does its conclusion derive? What are the credentials behind those citations and what are the details of those credentials? Again, the more that you intend to meaningfully rely upon ChatGPT’s outputs, the more skeptical you should be of its results until they are verified. Expertise and experience matter, so when using this tool it is important to understand the quality of its source material.

Numbers, Names, and Nouns

I worked in the eDiscovery industry for almost twenty years. This is the business of using digital information, such as emails and texts, as evidence in legal proceedings. There were many best practices that we learned as we sought to find smoking-gun messages buried in millions and sometimes billions of emails. Amongst these lessons learned is that some elements of language are more useful than others.

Numbers, Names, and Nouns refers to the approach of using word types to preferentially search through large volumes of data. Numbers are highly useful as they can be verified easily and unambiguously. Asking ChatGPT how many restaurants there are in Paris will likely give you a much more accurate and succinct response than asking it what is the best restaurant in Paris. The number of restaurants is also easily verifiable by other means, thereby providing a good sanity check.

Names refers to using proper nouns as a means of verification. The more ChatGPT presents proper nouns in its responses, the more likely it is that its results are both accurate and meaningful. If you asked ChatGPT about a certain musician and in its response it mentions a reference to an article in Rolling Stone magazine, it is likely that this will be a good reference on this topic.

Finally, the presence of nouns is generally more useful than verbs. Verbs refer to actions, and many actions have multiple meanings or multiple interpretations. This is less common with nouns. If you mention “jumping” one person might think of a basketball player while another might think of a kangaroo and a third might think of a child skipping rope. But mention a “rock” and most people will have a similar idea of what is being discussed.

So, the rule of thumb here is as you assess results coming from ChatGPT, use Numbers, Names and Nouns as opportunities to verify the results you are receiving for both accuracy and appropriateness.

Less is More

In the first installment of this series on ChatGPT I mentioned the importance of Shannon’s Information Theorem in generating good results in analyzing unstructured data, such as text.  A critical principle of this theorem is that the less often a piece of information appears in what you are analyzing, the more valuable and meaningful that piece of information is. In many ways this is the opposite of the “popularity model” followed by ChatGPT which makes this principle very useful in checking ChatGPT’s results.

For example, let’s imagine that you knew nothing about hurricanes and asked ChatGPT to tell you about hurricanes.  The word “hurricane” might appear in the resulting response twenty or thirty times, but the word “Katrina” might appear only once or twice. Asking ChatGPT deeper questions about “hurricanes” may not be more enlightening. However, asking ChatGPT to tell you more about “Hurricane Katrina” will likely provide substantially more information, and deeper context.  Note too that in this example “Hurricane Katrina” is a name, or proper noun, thus satisfying the preceding rule of using Numbers, Names, or Nouns in your searches.

A great example of this effect comes from the infamous Enron bankruptcy case. In the millions of emails that were examined by regulators only a handful of them identified the fraud underlying the company’s collapse. These emails were associated with the code word “Jedi”, a proper noun. This word seemed extremely out of place and out of context, which is why it stood out to examiners. Once they searched in and around the use of this word, the entire fraudulent scheme was rapidly identified.

Hence, when using ChatGPT to answer a question or learn something new, use those words or terms that appear least in the responses to craft your subsequent questions. Dig deep into those rarer terms and you are likely to gain stronger insights, faster.

Response Drift

A further interesting aspect of how ChatGPT is architected is the feedback mechanisms it uses to continue learning from our queries. It is collecting new information all the time, but also keeps track and learns from what we ask of it. As such, the answer it provides to a particular question today may be different at some point in the future; potentially significantly so. If you are intending to use the technology for some persistent uses such as contract drafting or copywriting, it should be standard practice to ask the system to periodically review or update its prior contributions.

In this way, you can both keep such documents up to date more easily, and also ensure that you don’t fall into copywrite or plagiarism situations. As more and more corporations subscribe to and utilize this platform there will be an ever-growing likelihood that people will start asking the same or similar questions, and hence will likely receive the same or similar answers. In many cases, this will be irrelevant. But, in those cases where such repetition might be a problem, it will likely be significantly so.

Your Friendly Neighborhood Cyborg

Truly effective use of a tool like ChatGPT should cause significant changes in how your organization operates. If it doesn’t, it’s not transformational. Such change is inherently disruptive, hence if you are not experiencing some degree of organization and managerial discomfort from adopting artificial intelligence you’re simply not trying hard enough. While achieving a new operational state where humans and bots work interdependently will not come easily nor cheaply, you will know you have succeeded when despite the effort and pain of change, you wouldn’t want to return to how things used to be.

Part 1: Chatpocalypse Now