Building a Conversational AI Assistant for Heavy Machinery

Modern heavy machinery is packed with features, making operation and maintenance increasingly complex. What if AI could make them easier to use? This article explores how AI assistants can revolutionize the way we interact with these complex machines. Discover the key considerations for integrating AI and how to avoid common pitfalls.

If you’re exploring the integration of AI assistants into your machinery – be it for manufacturing, maintenance, operation, or resale purposes – you’re likely considering the following questions:

• Should you build the solution in-house?
• Could a generic, off-the-shelf AI assistant suffice?
• Are there specialized solutions tailored to the specific demands of your industry?

We've spent a lot of time digging into this, talking to dozens of companies just like yours, and we've started to see some clear patterns emerge.

The Illusion of the Quick Prototype

Often, the first step is to bring in the experts – consultants or new hires – and start building a quick prototype using readily available libraries. You will most likely see some promising results early on with the system responding adequately to basic inquiries. However, as the project continues to the next phase, a more in-depth analysis often reveals the necessity for a highly specialized solution. What begins as a promising pilot project evolves into a massive, multi-year undertaking with no guaranteed finish line.

Operator manuals run to hundreds, sometimes thousands of pages, especially if maintenance instructions are included. Even with constantly increasing context windows, LLMs still struggle with this vast amount of information. Tests have shown that understanding drops off sharply as the text gets longer leading to hallucinated answers.

To avoid hallucinations, it is crucial to limit the amount of data that is fed to the LLM by feeding only the information that is relevant to the query. RAG (Retrieval-Augmented Generation) is a technique that has been developed to address this problem. Here’s how the most basic RAG pipeline works:

Basic RAG pipeline

Developing something like this will take anywhere from two to twelve weeks and will produce some nice early results where many responses are usually correctly derived from the relevant documentation. However, once you begin field-testing, you’ll soon find yourself back at the drawing board.

First field test with a naive RAG implementation

Retrieving and sharing correct textual information with users is only the first of many challenges ahead. And even this initial step can be unreliable if you rely on a naive RAG implementation!

Some of the typical complaints

Going Beyond the Basic Retrieval

Traditional search methods fall short when it comes to extracting the rich, multifaceted information contained within operator manuals. To truly leverage the power of AI and move beyond traditional paper manuals, we need a solution capable of intelligently accessing and presenting a wide range of elements, including:

• Images, tables, and diagrams: These elements need to be provided in the operator’s language with proper titles and legends
• Warnings and safety notices: Accurate retrieval of these notices is critical for preventing accidents and mitigating liability
• Step-by-step procedures: Especially for maintenance tasks, the AI should guide users through each step in a clear and structured manner. Ideally, completion data should also be recorded for maintenance logging purposes.
• Error codes, part numbers, and similar data: Basic semantic vector search is terrible at finding this type of information
• Industry-specific terminology: The AI must be able to understand and use the specialized vocabulary used in different industries and languages
• Machine-specific options and add-ons: The system should be configurable to reflect the unique specifications of each individual machine
• Links to external sources: Integration with relevant catalogs, videos, and other external sources enhances the user experience

To replace paper manuals, an AI assistant needs to pull all these elements together in a way that's easy to use. Because this is so specific to machinery, you won't find a generic AI platform that can do it all.

Implementing and extending full functionality of operator’s manuals

Designing an AI Assistant People Will Actually Want to Use

An AI assistant's true value isn't just about processing information; it's about delivering that information in a way that's intuitive, accessible, and useful. Users already have expectations based on other technologies – they want seamless, natural interactions that empower them to solve problems quickly. A poor user experience can render even the most advanced system useless. To ensure that the AI assistant truly changes how users interact with machinery documentation, these items must be prioritized:

• Natural communication: Users expect hands-free, conversational voice interaction using casual, everyday language rather than technical or "machine-friendly" prompts
• Comprehensive language support: Multilingual support for all forms of content - text, audio, image captions and more
• Accessible across devices: The interface must work seamlessly on phones, tablets, and desktop apps
• Familiar advanced features: Users increasingly expect access to advanced capabilities that they have encountered in other AI applications, such as:
  • Image-based input, such as snapping a photo of a warning light
  • Long-term memory of previous interactions
  • Integration with external APIs (e.g., weather queries)
  • Community notes: Allow users to contribute insights and expand the knowledge base through comments and shared experiences

As user familiarity with AI technologies grows, these expectations will continue to evolve. Proactively incorporating these features is paramount to creating a truly valuable, engaging, and user-friendly experience.

Not just a talking PDF – users will expect advanced features

The Quest for Accuracy

All the bells and whistles in the world mean nothing if the AI assistant isn't accurate and reliable. Accuracy is a challenge that all AI assistant solutions are currently struggling with. Right now, there's no magic bullet. The most dependable results come from meticulously optimizing the system for the specific task at hand, and that demands focused effort in these key areas:

• Effective chunking: Creating a strategic approach for breaking down the data is important and must be considered carefully when multimodal information is present
• Relational data mapping: The relationships between chunks of data need to be collected and stored, potentially using graph-based systems
• Optimized prompts: Tailoring prompts for each specific use case makes a big difference in the accuracy of the solution
• Latency management: The time between prompt and response must be kept low, especially for multi-step (agentic) workflows
• Observability and iteration: There must be a focus on continuous monitoring, and the use of user feedback and automated response quality analysis to improve performance
  

Excerpt from an analysis report from a field test conducted with real users

Beyond the specific challenges of AI, a robust engineering foundation is crucial for any successful AI assistant. This means focusing on core principles that ensure stability, adaptability, and security. Key considerations include:

• Adaptable architecture: Given the rapid pace of the AI industry, the solution needs to be flexible and ready to incorporate new innovations as they emerge
• Scalability: The solution needs to rapidly adapt to increasing traffic and user demand, ensuring consistent performance even during peak times
• Secure and compliant foundation: The system must be secure, reliable, and resilient by design, with strong user management, robust monitoring, and effective alerting, alongside the implementation of comprehensive data security measures and compliance with GDPR to ensure user data privacy

In our office, this monitoring dashboard is constantly displaying real-time performance data, ensuring immediate visibility into any deviations

Bringing It All Together

Witnessing the difficulties encountered by various organizations in adapting generic AI solutions to the unique demands of the heavy machinery sector, our team set out to create a purpose-built product. Our goal was to provide a comprehensive solution that addresses the specific challenges of this industry, preventing the reinvention of the wheel with each new implementation.

Example of a typical interaction with spogen.ai assistant:

1. The user asks for daily maintenance procedures

2. The assistant provides a spoken and visual (images and text) response

3. The user can follow step-by-step instructions with completion logging

4. Related warnings are easily accessible

While generic AI platforms offer a starting point, they lack the specialized knowledge and capabilities required to truly excel in the heavy machinery sector. Building an in-house solution to bridge that gap is a monumental undertaking. At spogen.ai we strive to offer a solution that understands the nuances of your industry.