AI-Based Natural Language Processing: Beyond Chatbots for Enterprise Solutions
Agenda
Computers and Natural Language
Dating back to the 1950s, the idea of processing natural human language with computers is almost as old as the modern computer itself. However, progress was slow, and results were underwhelming for a long time. While computers revolutionized signal processing in science and engineering and structured business data in the corporate world, natural language remained elusive for decades.
The tide began to turn when the field reluctantly shifted towards neural networks in NLP in the 2010s. For decades, language was processed by computers "symbolically," analogous to linguistic theory: Texts were split into sentences, sentences into words, and words into morphemes; grammar was represented by neatly organized trees. This makes intuitive sense, resembling the textbooks we know from school.
However, neural networks in NLP, based on long lists of numbers processed by multiplying with even longer tables of numbers, initially seemed an unlikely fit for language. Nevertheless, combining this approach with the novel Transformer architecture and massive training data brought about the long-elusive breakthrough in AI-based Natural Language Processing. Remarkably, these breakthrough algorithms are completely devoid of linguistic theory—no grammar trees, not even a traditional notion of a "word" exists in these systems.
This revolution is still fresh: The Transformer architecture in NLP was introduced in 2017, and the GPT-3 language model—which brought global attention to AI-based Natural Language Processing—was released in 2020.
NLP in the Enterprise
GPT and its successors have turned AI-based Natural Language Processing (AI-NLP) from a niche technology into a must-have capability with enormous commercial potential in the enterprise world. Below, I’ll provide an overview of how AI-NLP in business can be utilized across various industries.
Before diving into the specifics, it’s essential to understand some key principles from a business perspective:
Value Creation through Automation
The core mechanism of value creation with AI-NLP is automation: Computers can now perform tasks previously exclusive to humans, with sufficient quality and at scale. However, "at scale" with AI-NLP is significantly smaller compared to traditional data processing due to its high computational costs.
The degree of automation can vary from fully replacing human workers to employing "human-in-the-loop" approaches, where AI serves as an assistant. The choice depends on the required quality of the output. High-quality demands often necessitate more human involvement.
AI-NLP Core Capabilities
To maximize the benefits of AI-based Natural Language Processing, it is crucial to understand its core capabilities, which extend far beyond chatbots. These capabilities serve as building blocks for complex enterprise AI solutions.
Key AI-NLP Capabilities
Dialogue: Engaging in a plausible dialogue considering context and answering follow-up questions is probably the best-known capability of AI-NLP models, thanks to the popularity of ChatGPT. Alone, dialogue capability is not particularly useful, but it enables powerful applications when combined with other capabilities.
Question answering: AI-NLP can answer questions about a given piece of text in natural-language. They can also give answers to a large body of common knowledge questions in many fields, based on their training on massive amounts of data, which includes, for example, Wikipedia. Despite the professional tone, the answers should never be taken at face value as they can contain false statements, a behaviour known as hallucinations.
Implicit and explicit translation: AI-NLP can translate between languages. The quality typically depends on how much training data was available for the languages to be translated when the AI model was trained; hence it is best for English and considerably worse for rare languages. Many of the capabilities described here work across different languages (implicit translation). Question answering, for example, also works if the model is instructed to answer the question in a different language.
Summarisation and rewriting: AI-NLP can summarize text or rewrite it, modulating its style. This can be used, for example, to make a text less formal, or to make it sound as if written by a native speaker.
Text generation: AI-NLP can generate novel pieces of text in a desired style based on an instruction. This can be used to create marketing copy, product descriptions, documentation, social media posts or generic essays. The results are often well sounding but highly unoriginal content-wise. They many also contain false statements (hallucinations).
Text classification: AI-NLP can classify text into pre-defined categories. A classic example is sentiment analysis, where a product review or a social media post is determined to be either positive or negative. Other examples would be to classify the intent of a customer email (“billing issues”, “product help”, “address change”, etc.), to detect inappropriate content in user-generated text, or to determine a product category based on a product description in an online shop.
Information retrieval: AI-NLP can vastly improve information retrieval from large bodies of text. Classic information retrieval, as used by internet and intranet search engines, is based on keywords and simple text similarity scores. AI-based search is capable of capturing and matching meaning even if the query does not match a document on a keyword level. AI-based search also works across languages.
Information extraction:AI-NLP systems can not only generate answers for humans, but also structured output to be processed by machines or stored in databases. For example, assume we have a set of real-estate exposés with a lot of prose as PDF files. Using AI-NLP, we can generate a database table or an Excel file listing attributes like address, size, number of rooms for each property as columns. This information can be further processed to generate statistics, display information on a map, etc.
Function calling: This is a special capability of some models which allows an AI-NLP model to use other computer programs to use a task. The model is provided with a textual documentation of several “functions” (programs) it can use to solve a given task, and the information required to run these programs. It will answer which function to use to solve the task at hand, and which input to provide. An example of a function could be “Look up a customer’s order status using their email address”.
Coding: As a special case of text-generation, AI-NLP models can generate code in many programming languages. Like natural language answers containing false information, the generated code may or may not be correct. AI-generate code is also at risk of introducing security issues into a codebase and should therefore not be used unchecked. Analogous to question-answering, the models can also answer questions about existing code, for example, to explain errors. In this article, we won’t go deeper into this special domain.
Example: AI-NLP in Customer Support
Let’s take a real-world example of how these building blocks can be combined in a ticket-based customer support system, which blends AI and human support.
- Text Classification: First, the support ticket is classified into categories such as "product help," "order status," or "billing issues."
- Information Retrieval: For "product help" tickets, AI retrieves relevant documents from the company’s knowledge base.
- Information Extraction: The AI identifies the product in question by extracting relevant details from the user’s inquiry.
- Question Answering: Using the retrieved documents, the system automatically answers the customer’s questions.
- Implicit Translation: The AI responds in the customer’s language, even if the knowledge base is in another language.
- Dialogue: The system encourages follow-up questions and continues the conversation, while assessing whether the customer is satisfied or frustrated.
- Human Intervention: If the AI detects frustration or unhelpful answers, it passes the conversation to a human agent.
Conclusion: Maximizing AI-NLP in Business
To make the most out of AI-based Natural Language Processing in business, understanding its diverse capabilities is essential. Enterprise AI solutions should creatively combine these capabilities to solve business problems, carefully considering where human involvement is required.
