close button

A Guide to Navigating AI ChatBots in the Market Today Using the Evaluation Framework



Organizations increasingly use AI Chatbots to engage with customers and support staff in performing repetitive tasks more efficiently. For this CTO Corner, I have invited Dr. Martie Sablatura, and Mr. Andrew Hartigan, experts from ITCs technical team, to assess some of those popular chatbots used in the industry today.

Chatbots have been around for many years, but recently they have gained more popularity and awareness due to OpenAI ChatGPT. While chatbots were initially introduced to serve as service desk agents on predictable and easy-to-do tasks, technological advancements and generative AI have given rise to chatbots that make far more complex decisions. Essentially, chatbots are a piece of software that mimics human conversation using text, audio, or other digital formats. The benefits of this technology are being seen across industries offering time savings to teams to focus on higher priorities. Commonly, chatbots allow for personal experiences by answering common customer questions, collecting feedback, providing multilingual support to customers, and providing a consistent customer experience.

While many chatbot technologies exist, we sought out the most popular and most used out-of-the-box virtual agent solutions available at an enterprise level today. We briefly analyzed six offerings currently available to the public by reviewing the company’s websites, product documents as available, user feedback, or testimonials. We selected the following chatbot technologies due to their size, the companies’ involvement with maturing this technology, client requirements for developing their Chatbot performance criteria, and positive feedback from current users. We assessed the following six chatbot technologies in Exhibit 1 below.

Exhibit 1. Chatbot Technologies Assessment of Popular Tools Used Today

Current chatbot technologies and tools offer many features and functions. We developed an evaluation framework to meet our customers’ most relevant needs. You can leverage this framework to evaluate other chatbot technologies or assign different weights to each evaluation criterion per your organization’s specific requirements. For simplicity, we have not assigned any weights to our evaluation criteria; all criteria can be assumed to carry equal weight. Our evaluation framework consists of five criteria, as shown in Exhibit 2 below.

Exhibit 2. ITC Federal Chatbot Evaluation Framework

  1. Ease of Use –  Our experts reviewed the chatbot technologies to see how easily developers can implement the product, and users are able to interact with the chatbot to have their questions or requests.
  2. Scalability – Our experts assessed if the chatbots could handle high volumes of users and respond to customer requests and questions.
  3. Low Implementation and Maintenance Costs – Our experts assessed the maintenance, updates, and revisions required to support the organization’s needs over time and the cost to maintain the technology.
  4. Security –  Our experts reviewed the security processes in maintaining the chatbot and authentication, verification, and authorization needs.
  5. Natural Language Understanding (NLU) – Our experts reviewed to see how well the chatbot interacts with the user and understands inputs using speech and text requirements without any limitations.

The following sections provide a broad assessment of these chatbot technologies against the devised evaluation framework and the resulting assessment with their advantages and other considerations.

With the pace of innovation, chatbot vendors are continuously adding new features, and the following assessment is relevant based on the technologies presented currently. We encourage readers to consult the provided links for each chatbot provider for more of the latest information and features as time goes on. We provide this assessment only as guidance as your development teams assess the chatbot technologies that are right for your organization.

IBM Watson Assistant

IBM’s Watson Assistant is an Artificial Intelligence (AI) product that allows developers to build, train, and deploy conversational interactions into any application, device, or channel. It can be deployed in any cloud or on-premises environment and is considered an industry-leading NLU tool. It provides users with training recommendations on building, running, and improving their Assistant. The tool also knows when to search for an answer from a knowledge base, ask for clarity, and direct users to a human. Highlighted in Exhibit 3 are some advantages and things to take into consideration when assessing the Watson Assistant to support your IT need:

Exhibit 3. Advantages and Considerations of Using the Watson Assistant

Microsoft Bot

The Microsoft Bot Framework is a collection of libraries, tools, and services that allow developers to build, test, deploy, and manage intelligent bots. The Bot Framework includes an extensible Software Development Kit (SDK) for building bots and connecting to AI services. With this framework, developers can create bots that use speech, understand natural language, answer questions, and more. Additionally, developers can choose their favorite development environment or command line tools to create a bot (i.e., C#, Java, JavaScript, TypeScript, and Python). Highlighted in Exhibit 4 are some advantages and things to consider when assessing the Microsoft Bot to support your IT need:

Exhibit 4. Advantages and Considerations of Using the Microsoft Bot

Microsoft Azure Bot Services

Azure Bot Services integrates powerful AI capabilities with Azure Cognitive Services to keep pace with technological advances without a machine learning experience. This service enables developers to build a basic informational bot to alleviate users’ high volume of common inquiries with an open and extensible bot framework. Then, as you learn more about the usage, you can create more advanced bots, including virtual assistants that can handle rich, multiturn conversations that make it easier to engage with your users. You can deliver personalized experiences to customers without compromising ownership and control of customer data and deploy a virtual assistant tailored to your brand, across multiple apps and devices, with the Virtual Assistant solution accelerator. Highlighted in Exhibit 5 are some advantages and things to consider when assessing the Azure Bot to support your IT need:

 

Exhibit 5. Advantages and Considerations Using the AzureBot

Amazon Web Services (AWS) Chatbot

AWS Chatbot is an interactive agent that makes monitoring and interacting with your AWS resources in your Microsoft Teams or Slack channels easier. Using AWS Chatbot, you can receive alerts and run commands to return diagnostic information, invoke AWS Lambda functions, and create AWS Support cases so your team can collaborate and respond to events faster. AWS Chatbot processes AWS service notifications from Amazon Simple Notification Service (Amazon SNS) and forwards them to chat rooms so teams can analyze and act on them immediately, regardless of location. There are many different ways to integrate AWS Chatbot with other AWS services.

AWS Chatbot uses Amazon Simple Notification Service (Amazon SNS), a managed service that delivers messages from publishers to subscribers. Publishers communicate asynchronously with subscribers by sending messages to a topic, a logical access point and communication channel. To use this tool, you must have the following prerequisites to move forward with the following steps:

  • AWS Account – Used to support access to AWS Services and support in building and deploying AWS products.
  • Amazon GuardDuty – Amazon’s threat detection service must be enabled to support workloads and monitor threats and malicious activities.
  • Slack Account (Slack Workspace ID and Channel ID) – Access to communication platform with tips to help troubleshoot issues. Supports collaborative and communication spaces that allows teams to connect software tools and services.

Highlighted in Exhibit 6 are some advantages and things to consider when assessing the AWS Chatbot to support your IT need:

Exhibit 6. Advantages and Considerations Using the AWS Chatbot

ChatGPT

ChatGPT is an AI model that uses deep learning to generate human-like text based on user prompts. It predicts the next word in a given text based on the patterns it has learned from a massive amount of data during its training process. Although the core function of a ChatBot is to mimic a human conversationalist, ChatGPT is versatile and uses advanced natural language processing. Highlighted in Exhibit 7 are some advantages and things to consider when assessing the ChatGPT within your current IT environment:

Exhibit 7. Advantages and Considerations Using ChatGPT

Google’s Dialogflow

Google’s Dialogflow is an NLU platform that makes it easy to design and integrate a conversational user interface into your applications, tools, or devices. Dialogflow is a platform that can be used to design conversational AI interactions such as ChatBot, voice assistant, and virtual assistants, integrated with different conversational platforms, like Google Assistant. Dialogflow is a platform that enables the development of ChatBot without the need for installation, technical infrastructure, and coding knowledge. It is easy to integrate it into many different platforms directly without extra coding, and it can analyze multiple input types from your customers. Highlighted in Exhibit 8 are some advantages and things to consider when assessing Dialogflow:

Exhibit 8. Advantages and Considerations Using the Google Dialogflow

IBM Watson, Microsoft Bot, Azure Bot Services, AWS Chatbot, ChatGPT, and Dialogflow all offer advantages:

  • Advanced NLP: While ServiceNow’s virtual agent offers basic NLP capabilities, these other platforms use more advanced NLP algorithms, enabling them to understand better and respond to natural language input, resulting in a more accurate and effective user experience.
  • Customizable Functionality: IBM Watson, Microsoft Bot, AWS Chatbot, ChatGPT, and Dialogflow, offer a more comprehensive range of pre-built functionalities and integrations that can be customized to meet specific use cases, making it easier to handle more complex interactions and offers a personalized user experience. While ServiceNow’s virtual agent offers some customization options, they are more limited.
  • Industry Expertise: These platforms have been widely deployed across various industries and, as a result, have developed industry-specific expertise and best practices that can help deliver better outcomes. ServiceNow’s virtual agent primarily focuses on IT service management, which can limit its effectiveness in other business areas.

The advanced NLP capabilities, customizable functionality, and broader industry expertise of IBM Watson, Microsoft Bot, Dialogflow, AWS Chatbot, and ChatGPT can provide comprehensive chatbot solutions. This decision depends on the specific selection criteria on which option is the best for your customers or internal IT and business needs.

As a leading midsized federal systems integrator, ITC Federal brings the capabilities to help in assessing how chatbots can support organizations’ daily operations, from assisting staff with daily task activities to enhancing the customer’s experience. At ITC, we bring certified developers and technical experts who have been early adopters of the ChatBot technologies and offer expertise in Data Analytics; they can build interactive applications to enhance customers’ operational efficiencies. We can help ensure your company chooses the Chat Bot, AI, or Machine Learning technology to meet your desired business and IT outcomes.