While companies continue to invest heavily in technology like business intelligence (BI) and data integration tools in an effort to reign in their financial and other numerical data, a veritable gold mine of information remains virtually untapped in textual and unstructured data, such as e-mail, blogs and call center agent notes.
However, the market for text-aware applications remains fractured and confusing, according to a recent report from The 451 Group, an IT analyst firm in New York.
"Everybody in IT has something to gain or should be considering unstructured data analysis," said Martin Schneider, an analyst with the 451 Group and an author of the report. "There's been this disconnect between 'Is this to handle a specific industry issue like credit card fraud protection?' or something you apply to the whole enterprise, which can then become unwieldy and can get lost -- much the way people bought these huge CRM systems."
Similar to CRM, where companies began with "big-bang" all or nothing implementations and saw widespread failure, the report advises both vendors and users to take a more phased approach with text analysis.
IBM made its own contribution this week, making its Unstructured Information Management Architecture available to the open source community. Along with the release, IBM said it is enabling search technology that goes beyond keywords.
"Because this allows you to understand semantic meanings as well as queries, I can significantly improve search results, which is one of the biggest pain points in the enterprise," said Nelson Mattos, IBM distinguished engineer and vice president strategy, WebSphere Information Integration Solutions. "Employees spend a third of their time looking for documents they need to get their job done. The second biggest pain point is that 85% of the corporate data that exists in every company's computers is unstructured. You can use this technology to understand what those documents are telling you."
According to the 451 Group, the future of a large portion of the software market will depend on which companies manage to integrate unstructured data analysis into their own applications and deploy it first. However, most business applications vendors, such as those in BI and CRM, have failed to address unstructured data, and many organizations have ignored their problems with unstructured data.
"Call center documents, PowerPoint presentations, articles, scans of documents --enterprises do not have the technology to allow them to gain insight into this 85% of data," Mattos agreed. "This is huge competitive advantage for a company."
Some companies have begun to look at unstructured data. Fraud detection has become an early application of the technology by helping to determine unusual credit card buying patterns.
As has market insight. Companies are using technology to crawl through Web sites and message boards to get an unpolished, yet accurate, idea of brand awareness, Schneider said. One notable example is that of Volkswagen, which discovered that customers were complaining that they didn't have a way to connect their iPod digital music players into their car stereo and questioned the need for a six-disc CD changer. Volkswagen went out and ran a new promotion, tapping into the market.
"The CRM vendors can really stand to gain in adding this to their systems," Schneider said. "It hasn't been done on a large scale. What you're seeing are very localized products. The sales department might look for better forecasting, service for fault. You don't want the Firestone-Ford Explorer thing to happen again. To really tie this all together in a cohesive framework the way CRM grew into its own, to empower all of these areas with unstructured data, that's the next step."
Until that happens, technology buyers should proceed cautiously with investment, Schneider warned.
"Ask where are the pain points, where can I gain more value and more agility through analysis of text," he said. "How is text going to make me smarter, faster, better."