Why generative AI often fails to deliver value

Introduction: AI’s promise vs. reality

Artificial Intelligence (AI) has been heralded as a game-changer for businesses, promising to transform industries, boost efficiency, and unlock unprecedented innovation. Yet, a stark reality persists: while 98% of companies are exploring AI, only 26% have managed to move beyond pilot projects and generate real value. A recent report by Boston Consulting Group (BCG) sheds light on this gap, highlighting the barriers organizations face and what distinguishes successful AI adopters from the rest.

The common barriers to AI success

Despite significant investments, many organizations struggle to make AI work for them. BCG’s findings point to several key challenges:

1. Many companies treat AI as a buzzword, chasing too many projects without clear strategic alignment. This scattershot approach dilutes resources and results in underwhelming outcomes.
2. Organizations often fail to integrate AI into their existing processes, IT systems, and broader business strategies.
3. The most successful AI initiatives rely on a blend of technical expertise and organizational readiness. However, many businesses lack skilled AI professionals and struggle to cultivate AI literacy across their workforce.
4. While it’s easy to pilot an AI project, scaling it across an organization is a monumental task. Issues like inconsistent data quality, security concerns, and resistance to change can derail efforts.
5. A disproportionate focus on algorithms and technical tools often neglects the most critical components: people and processes. Without reimagining workflows and investing in change management, even the best AI systems can fail to deliver.

What successful companies do differently

According to BCG, the 26% of companies thriving with AI adoption share distinct characteristics that set them apart:

• They prioritize a handful of high-value initiatives that align with their core business objectives.
• Leaders allocate resources thoughtfully—70% to people and processes, 20% to technology, and just 10% to algorithms.
• They build infrastructure that supports scalable AI solutions while rapidly iterating on pilots to refine their impact.
• By investing in AI literacy and fostering a culture of innovation, they empower employees to embrace and leverage AI effectively.
• Beyond cost-cutting, these companies use AI to create new revenue streams, optimize core operations, and enhance customer experiences.

Managing expectations and addressing common barriers

From our experience working with clients, we see a mix of these common barriers highlighted by BCG. Two key challenges stand out: skills and process gaps and integration and scalability hurdles. These are critical areas where organizations often falter. A major factor slowing adoption is a lack of knowledge about the boundaries of AI systems. Generative AI applications are particularly susceptible to inflated expectations, with many customers anticipating near-perfect accuracy from the outset. However, due to the probabilistic nature of these systems, achieving a 100% success rate is inherently difficult. Mismanagement of these expectations can lead to frustration and disillusionment. Organizations must recognize that the first iteration of a GenAI application does not need to be perfect. Instead, the focus should be on delivering incremental improvements and building trust over time. That being said, there are several steps AI developers can take to make GenAI applications more reliable and trustworthy.

Making GenAI applications more trustworthy: Lessons from Applied LLMs

The Applied LLMs team provides several actionable strategies to address these challenges and enhance trust in AI applications:

• Break complex tasks into smaller, manageable components with clear objectives. For example, decompose a single large prompt into multiple focused steps, such as extracting key details, verifying them, and synthesizing a summary.
• Build rigorous unit tests and assertion-based validations to catch failure modes early. Evaluations should be specific to real-world use cases and evolve as the system improves.
• Focus on providing relevant, concise, and detailed documents for better outcomes. RAG helps ground responses in reliable data, reducing hallucinations and improving factual accuracy.
• Employ techniques like chain-of-thought reasoning and in-context learning to improve model performance. Use examples that reflect real-world distributions to guide the model.
• Design systems where users can validate and improve outputs. HITL not only enhances trust but also creates a feedback loop for continuous model refinement.
• Ensure that prompts are concise and free from unnecessary complexity. Overloading the model with irrelevant information can degrade performance.

By combining these strategies with realistic expectations, organizations can significantly improve adoption and trust in GenAI systems.

A balanced path to success

The journey to successful AI adoption requires a blend of technical excellence, realistic goal-setting, and continuous iteration. While challenges like integration and scalability hurdles persist, focusing on skills development, operational workflows, and trust-building measures can set organizations on the right path. Essentially, AI works best when technical accuracy is paired with a clear understanding of its limitations. By setting realistic expectations and focusing on building reliable and scalable systems, businesses can create value with AI.