Study Reveals Challenges at the Intersection of AI Innovation and Agile Methodologies: Insights From ZDNet

Maxger/Getty Images

Agile software development has long been seen as a highly effective way to deliver the software the business needs. The practice has worked well within many organizations for more than two decades. Agile is also the foundation for scrum , DevOps , and other collaborative practices. However, agile practices may fall short in artificial intelligence (AI) design and implementation.

Also: The most popular programming languages in 2024 (and what that even means)

That insight comes from a recent report by RAND Corporation, the global policy think tank, based on interviews with 65 data scientists and engineers with at least five years of experience building AI and machine-learning models in industry or academia. The research, initially conducted for the US Department of Defense, was completed in April 2024. “All too often, AI projects flounder or never get off the ground,” said the report’s co-authors, led by James Ryseff , senior technical policy analyst at RAND.

Interestingly, several AI specialists see formal agile software development practices as a roadblock to successful AI . “Several interviewees (10 of 50) expressed the belief that rigid interpretations of agile software development processes are a poor fit for AI projects,” the researchers found.

Also: Agile development can unlock the power of generative AI - here’s how

“While the agile software movement never intended to develop rigid processes – one of its primary tenets is that individuals and interactions are much more important than processes and tools – many organizations require their engineering teams to universally follow the same agile processes.”

Newsletters

ZDNET Tech Today

ZDNET’s Tech Today newsletter is a daily briefing of the newest, most talked about stories, five days a week.

See all

As a result, as one interviewee put it, “work items repeatedly had to either be reopened in the following sprint or made ridiculously small and meaningless to fit into a one-week or two-week sprint.” In particular, AI projects “require an initial phase of data exploration and experimentation with an unpredictable duration.”

Also: How your business can best exploit AI: Tell your board these 4 things

RAND’s research suggested other factors can limit the success of AI projects. While IT failures have been well documented over the past few decades, AI failures take on an alternative complexion. “AI seems to have different project characteristics, such as costly labor and capital requirements and high algorithm complexity, that make them unlike a traditional information system,” the study’s co-authors said.

“The high-profile nature of AI may increase the desire for stakeholders to better understand what drives the risk of IT projects related to AI.”

The RAND team identified the leading causes of AI project failure:

“Industry stakeholders often misunderstand – or miscommunicate – what problem needs to be solved using AI. Too often, organizations deploy trained AI models only to discover that the models have optimized the wrong metrics or do not fit into the overall workflow and context.”
“Many AI projects fail because the organization lacks the necessary data to adequately train an effective AI model.”
“The organization focuses more on using the latest and greatest technology than on solving real problems for their intended users.”
“Organizations might not have adequate infrastructure to manage their data and deploy completed AI models, which increases the likelihood of project failure.”
“The technology is applied to problems that are too difficult for AI to solve. AI is not a magic wand that can make any challenging problem disappear; in some cases, even the most advanced AI models cannot automate away a difficult task.”

While formal agile practices may be too cumbersome for AI development, it’s still critical for IT and data professionals to communicate openly with business users . Interviewees in the study recommended that “instead of adopting established software engineering processes – which often amount to nothing more than fancy to-do lists – the technical team should communicate frequently with their business partners about the state of the project.”

Also: Time for businesses to move past generative AI hype and find real value

The report suggested: “Stakeholders don’t like it when you say, ‘it’s taking longer than expected; I’ll get back to you in two weeks.’ They are curious. Open communication builds trust between the business stakeholders and the technical team and increases the likelihood that the project will ultimately be successful.”
Therefore, AI developers must ensure technical staff understand the project purpose and domain context: “Misunderstandings and miscommunications about the intent and purpose of the project are the most common reasons for AI project failure. Ensuring effective interactions between the technologists and the business experts can be the difference between success and failure for an AI project.”

The RAND team also recommended choosing “enduring problems”. AI projects require time and patience to complete: “Before they begin any AI project, leaders should be prepared to commit each product team to solving a specific problem for at least a year. If an AI project is not worth such a long-term commitment, it most likely is not worth committing to at all.”

Also: When’s the right time to invest in AI? 4 ways to help you decide

While focusing on the business problem and not the technology solution is crucial, organizations must invest in the infrastructure to support AI efforts , suggested the RAND report: “Up-front investments in infrastructure to support data governance and model deployment can substantially reduce the time required to complete AI projects and can increase the volume of high-quality data available to train effective AI models.”

Finally, as noted above, the report suggested AI is not a magic wand and has limitations: “When considering a potential AI project, leaders need to include technical experts to assess the project’s feasibility.”