By Rishab Narang (Curated from CSI's Big Data Edition)
Socio-Business Intelligence Using Big Data
Abstract: We describe how the fusion of social and business intelligence is defining the next-generation of business analytics applications using a new AI-driven information management architecture that is based on big-data technologies and new data sources available from social media.
What is ‘BigData’?
The term ‘BigData’ has become the latest buzzword in the IT industry, much as Cloud Computing began to elicit interest a few years ago. As in the case of the latter, we submit that BigData, is a metaphor for a few significant technology, social business convergences: Popular interest in cloud computing was fuelled by the emergence and eventual confluence of web-based social applications, software as a service, infrastructure as a service, and finally platforms as a service. In a similar fashion, ‘BigData’ is essentially the convergence of technology advances in artificial intelligence emanating from search and online advertising, along with the development of new architectures for managing extremely large web-scale data volumes, exemplified by the now popular Hadoop stack. Along with the means to process vast quantities of unstructured data, we also find that the data itself is now readily available: Vast volumes of consumer conversations on social media;, such as Twitter, are free for all to access, and the rest are rapidly becoming a valuable commodity available for purchase from Facebook, LinkedIn, etc. In this article we describe a number of ‘Socio-Business’ applications that exploit these new data sources, and are of potential interest to large enterprises. Moreover, we find that each of these applications involve the fusion of information from social media with internal business data, the extraction of knowledge from web-sources, the application of artificial intelligence techniques in some fashion, and/or the exploitation of BigData-inspired data- management architectures.
The New Context for Business Intelligence In the past decade, AI techniques operating at web-scale have now demonstrated significant successes on the web, many of which were once impossible: Statistical machine learning at web-scale is the reason, why Google’s machine translation works. Web-based face recognition relies, among other things, on large-scale multi- way clustering to discover image features that work best to disambiguate faces; this couple with some human tagging, even from profile photos, is then sufficient to recognise faces even without standard scale, poses, expression or illuminations. The Watson system uses 15 terabytes of in-memory data culled from the web and other sources, along with parallel processing across 90 processors. Finally, Siri’s hope for success depends on the fact that it includes a cloud-component, which opens up the possibility for continuous learning using the large volumes of data its adoption my millions of users will generate. The time is therefore ripe for enterprise business. AI techniques into their solutions. The potential for AI techniques in the enterprise was aptly articulated almost a decade ago by Dalal et. Al. Moreover, the availability of large volumes of data from social-media makes it all the more viable, as well as essential to exploit the techniques already being used so well in web-scale AI applications. Further, in sharp contrast to the millions of servers powering the web, the largest of enterprise IT departments, are used to handling 50,000 or so servers, and hundreds of terabytes of data at the most. Enterprise data-storage, databases, and data analysis tools are, in turn, tailored to handle terabytes or at most a petabyte or so. Further, most of the ‘big-data’ emanating from social-media sources is unstructured, text data; again, something that the traditional business intelligence tool-stack is not designed to tackle, and for which before mentioned AI techniques, are needed to extract insight. Moreover, inputs from social media comprises of largely unstructured data that needs to be tapped, processed and analysed sometimes require the use of big-data technologies such as are used by the web companies themselves, instead of the traditional databases that are better suited for structured data. Thus, big-data technologies such as Hadoop etc. are often used even though most traditional enterprises do not actually need to process as large a volume of data as the web companies do. Innovative business use-cases exploiting BigData from social-media and mobility sources span multiple industries, from retail to manufacturing and financial services. A common theme across all these applications besides having to extract intelligence, from large-volumes of BigData is the need to fuse information from multiple sources, internal and external, structured and unstructured. Further, the rapid pace of developing events on social media mean that the standard techniques for translating predictive insights into real-time decision support, such as building (off-line) a deep but computationally ‘small’ model, need to be enhanced: Social media events need to be filtered, processed, correlated and analysed for their impact in real-time. In the sections that follow, we describe some of these use-cases, and explain the techniques they require.
Supply-Chain Disruptions
he recent natural disasters that struck Japan in 2011, i.e., the earthquake, tsunami, and subsequent release of nuclear radiation, clearly had a devastating effect on the Japanese population and economy. At the same time, the effects of these events were felt around the world; in particular, they led to major disruptions in the global supply chain for many industries, from semiconductors to automobiles and even baby products. The Japanese earthquake was a major event of global significance, followed closely in the global media on a daily basis; hopefully a fairly rare ‘black swan’ event. However, many adverse events of a far smaller significance occur daily across the world. Such events are mainly of local interest only. Further, public interest in the event may last but a day or so, while its economic impact may last much longer: Take the example of a fire in a factory. There are, on the average, in the range of ten or so major factory fires in the world every day. Similarly there are labour strikes that disrupt production. Most of these events affect a very small locality, and may not even reach the local news channel, and certainly not global ones. Further, any public interest, however localised, in the event may last a few hours or at most a day. Nevertheless, if the factory affected is a significant supplier to a major manufacturer half-way around the world, this relatively minor event is possibly of great interest to the particular enterprise that consumes its product! It is observed that the manufacturers notice such news about their suppliers, when they encounter shortage in supply, which is usually a few days or sometime a week later. If however technology can help them notice this earlier, they will have more time to make alternate arrangements. Interestingly, it has been found that many of these events, even the ones with extremely local impact, find their way fairly rapidly into social media, and in particular Twitter. Used for social-networking in over 200 countries with over 500 million tweets a day, Twitter turns to also be rich source of local news from around the world. Many events of local importance are first reported on Twitter, including many that never reach news channels. In [5] we have proposed an architecture that enables a large enterprise to monitor potential disruptions in its global supply-chain by detecting adverse events, by monitoring Twitter streams. In [4] we have described how such events can be efficiently detected using machine- learning techniques, from amongst streams of unstructured short-text messages (tweets) arriving at a rate of tens of messages per second. In contrast with the larger volumes that follow events of wider significance, there are often only a few tweets reporting each such event; the few tweets that happen to report the same event, are then correlated. Next, as described in[5] and, the impact of the detected event to the enterprise in question can be assessed, by fusing the detected external event with internal data on suppliers.
Voice of the Consumer Listening to the voice of the consumer through mechanisms such as surveys, feedbacks, emails, and support center logs, is a continual process through which organizations try to improve their customer satisfaction rate and increase their consumer base. Increasingly, listening to consumer-generated content from social-media channels like Twitter, Facebook, and Blogosphere is augmenting the possibilities for analyzing the voice of the consumer, and becoming an important element of the business intelligence strategy of consumer-focused enterprises. At the same time, the traditional channels of listening directly to customers, such as call-centers and email, and indirectly through eventual sales figures, remain as important as ever: Social-media inputs are inherently noisy in nature, so the insights acquired from social-media are often validated by fusing these with additional inputs collected through more traditional channels. At the same time, social-media inputs may often lead other inputs in time, and therefore be of significance in spite of their relative inaccuracy. Different type of insights can be gathered from consumer- generated content. Companies are engaging in analyzing the voice of consumer primarily to address the following issues, which we may also distinguish based on the the content, sources, and temporal variation that they focus on:
1. Brand Sentiment Analysis, is concerned with measuring the sentiment expressed in the context of particular brands, products, and services, or even specific pre-defined features of a product or service. The emphasis is on volumes, and on tracking the overall aggregate positivity / negativity associated with the set of concepts one is interested in. Source selection is broad and channel based; thus one might choose to focus on say Twitter, a Facebook page, and selected blogs, as well as analyze the variation across these. Since sentiment is noisy and varies rapidly, it is also aggregated temporally; thus the time- scales of aggregate sentiment analysis are in the range of days and weeks. Social-media-based brand sentiment analysis is cheaper and faster than traditional survey-based techniques such as Nielsen market-surveys; it also reveals results sooner. Thus, sudden and significant changes in sentiment about a brand can be detected faster, such as that which took place when Tropicana changed its packaging a few years ago, which was followed by strong negative consumer sentiment. However, the jury remains out as to how often these aggregate sentiment figures bring novel insights as compared to traditional measures. The fact is they need to be time-averaged to make any sense; thus finer-grained approaches are needed to enable more real-time response, and detect emerging problems that by themselves may not change the aggregate sentiment significantly, at least at first, and that too only if not addressed in time. Listening to consumer sentiment on social platforms has recently become almost a commodity offered by a number of commercial services, such as Radian61 and others. Opinion-mining techniques for extracting sentiment from text are used in such tools. The initial insight that is most often sought through the adoption of a listening service is the ability to monitor brand perception, i.e., whether consumers at large are saying positive or negative things about one’s brand, product, or service.
2. omplaint Analysis, in contrast with brand sentiment analysis that casts its net wide, complaint analysis tries to focus on actual customers. Thus, the sources for such analysis are either direct customer feedback through call-centers or email, or when it comes to social-media, the input is carefully filtered so as to ensure the presence of indicators such as “I bought”, “my car”, etc., making it highly likely that the writer is in fact a customer, either of one’s own product or that of a competitor. Next, such complaint analysis aims to analyze the text written by customers to detect which aspects of a product or service they are having difficulty with. This requires a deeper level of natural language processing than, say, aggregate sentiment analysis: Consider the statement “I’ve been having trouble with my new [car- brand], not only did the transmission give way in the first month but there was a significant delay in getting it changed”. Clearly it is a negative statement about the car brand, and even its transmission, which basic sentiment analysis can easily discover. However, deeper text processing can further discern what exactly is wrong with the transmission, and aggregate such difficulties across a large volume of customer feedback along various dimensions. As a result, if the concept of say, transmission ‘giving way’ including its linguistic equivalents, is showing up in significant numbers, then this becomes an issue to flag to product engineering. On the other hand, the fact that the supply of spares of various types are delayed, including transmission parts etc., gets aggregated at a different level of say ‘delayed parts’, and is escalated to those responsible for after-sales services. The deeper degree of text processing required for complaint analysis requires ‘ontology-driven causal analysis’, which involves some level of parsing as well as learning, and exploiting domain ontology.
Abstract: We describe how the fusion of social and business intelligence is defining the next-generation of business analytics applications using a new AI-driven information management architecture that is based on big-data technologies and new data sources available from social media.
What is ‘BigData’?
The term ‘BigData’ has become the latest buzzword in the IT industry, much as Cloud Computing began to elicit interest a few years ago. As in the case of the latter, we submit that BigData, is a metaphor for a few significant technology, social business convergences: Popular interest in cloud computing was fuelled by the emergence and eventual confluence of web-based social applications, software as a service, infrastructure as a service, and finally platforms as a service. In a similar fashion, ‘BigData’ is essentially the convergence of technology advances in artificial intelligence emanating from search and online advertising, along with the development of new architectures for managing extremely large web-scale data volumes, exemplified by the now popular Hadoop stack. Along with the means to process vast quantities of unstructured data, we also find that the data itself is now readily available: Vast volumes of consumer conversations on social media;, such as Twitter, are free for all to access, and the rest are rapidly becoming a valuable commodity available for purchase from Facebook, LinkedIn, etc. In this article we describe a number of ‘Socio-Business’ applications that exploit these new data sources, and are of potential interest to large enterprises. Moreover, we find that each of these applications involve the fusion of information from social media with internal business data, the extraction of knowledge from web-sources, the application of artificial intelligence techniques in some fashion, and/or the exploitation of BigData-inspired data- management architectures.
The New Context for Business Intelligence In the past decade, AI techniques operating at web-scale have now demonstrated significant successes on the web, many of which were once impossible: Statistical machine learning at web-scale is the reason, why Google’s machine translation works. Web-based face recognition relies, among other things, on large-scale multi- way clustering to discover image features that work best to disambiguate faces; this couple with some human tagging, even from profile photos, is then sufficient to recognise faces even without standard scale, poses, expression or illuminations. The Watson system uses 15 terabytes of in-memory data culled from the web and other sources, along with parallel processing across 90 processors. Finally, Siri’s hope for success depends on the fact that it includes a cloud-component, which opens up the possibility for continuous learning using the large volumes of data its adoption my millions of users will generate. The time is therefore ripe for enterprise business. AI techniques into their solutions. The potential for AI techniques in the enterprise was aptly articulated almost a decade ago by Dalal et. Al. Moreover, the availability of large volumes of data from social-media makes it all the more viable, as well as essential to exploit the techniques already being used so well in web-scale AI applications. Further, in sharp contrast to the millions of servers powering the web, the largest of enterprise IT departments, are used to handling 50,000 or so servers, and hundreds of terabytes of data at the most. Enterprise data-storage, databases, and data analysis tools are, in turn, tailored to handle terabytes or at most a petabyte or so. Further, most of the ‘big-data’ emanating from social-media sources is unstructured, text data; again, something that the traditional business intelligence tool-stack is not designed to tackle, and for which before mentioned AI techniques, are needed to extract insight. Moreover, inputs from social media comprises of largely unstructured data that needs to be tapped, processed and analysed sometimes require the use of big-data technologies such as are used by the web companies themselves, instead of the traditional databases that are better suited for structured data. Thus, big-data technologies such as Hadoop etc. are often used even though most traditional enterprises do not actually need to process as large a volume of data as the web companies do. Innovative business use-cases exploiting BigData from social-media and mobility sources span multiple industries, from retail to manufacturing and financial services. A common theme across all these applications besides having to extract intelligence, from large-volumes of BigData is the need to fuse information from multiple sources, internal and external, structured and unstructured. Further, the rapid pace of developing events on social media mean that the standard techniques for translating predictive insights into real-time decision support, such as building (off-line) a deep but computationally ‘small’ model, need to be enhanced: Social media events need to be filtered, processed, correlated and analysed for their impact in real-time. In the sections that follow, we describe some of these use-cases, and explain the techniques they require.
Supply-Chain Disruptions
he recent natural disasters that struck Japan in 2011, i.e., the earthquake, tsunami, and subsequent release of nuclear radiation, clearly had a devastating effect on the Japanese population and economy. At the same time, the effects of these events were felt around the world; in particular, they led to major disruptions in the global supply chain for many industries, from semiconductors to automobiles and even baby products. The Japanese earthquake was a major event of global significance, followed closely in the global media on a daily basis; hopefully a fairly rare ‘black swan’ event. However, many adverse events of a far smaller significance occur daily across the world. Such events are mainly of local interest only. Further, public interest in the event may last but a day or so, while its economic impact may last much longer: Take the example of a fire in a factory. There are, on the average, in the range of ten or so major factory fires in the world every day. Similarly there are labour strikes that disrupt production. Most of these events affect a very small locality, and may not even reach the local news channel, and certainly not global ones. Further, any public interest, however localised, in the event may last a few hours or at most a day. Nevertheless, if the factory affected is a significant supplier to a major manufacturer half-way around the world, this relatively minor event is possibly of great interest to the particular enterprise that consumes its product! It is observed that the manufacturers notice such news about their suppliers, when they encounter shortage in supply, which is usually a few days or sometime a week later. If however technology can help them notice this earlier, they will have more time to make alternate arrangements. Interestingly, it has been found that many of these events, even the ones with extremely local impact, find their way fairly rapidly into social media, and in particular Twitter. Used for social-networking in over 200 countries with over 500 million tweets a day, Twitter turns to also be rich source of local news from around the world. Many events of local importance are first reported on Twitter, including many that never reach news channels. In [5] we have proposed an architecture that enables a large enterprise to monitor potential disruptions in its global supply-chain by detecting adverse events, by monitoring Twitter streams. In [4] we have described how such events can be efficiently detected using machine- learning techniques, from amongst streams of unstructured short-text messages (tweets) arriving at a rate of tens of messages per second. In contrast with the larger volumes that follow events of wider significance, there are often only a few tweets reporting each such event; the few tweets that happen to report the same event, are then correlated. Next, as described in[5] and, the impact of the detected event to the enterprise in question can be assessed, by fusing the detected external event with internal data on suppliers.
Voice of the Consumer Listening to the voice of the consumer through mechanisms such as surveys, feedbacks, emails, and support center logs, is a continual process through which organizations try to improve their customer satisfaction rate and increase their consumer base. Increasingly, listening to consumer-generated content from social-media channels like Twitter, Facebook, and Blogosphere is augmenting the possibilities for analyzing the voice of the consumer, and becoming an important element of the business intelligence strategy of consumer-focused enterprises. At the same time, the traditional channels of listening directly to customers, such as call-centers and email, and indirectly through eventual sales figures, remain as important as ever: Social-media inputs are inherently noisy in nature, so the insights acquired from social-media are often validated by fusing these with additional inputs collected through more traditional channels. At the same time, social-media inputs may often lead other inputs in time, and therefore be of significance in spite of their relative inaccuracy. Different type of insights can be gathered from consumer- generated content. Companies are engaging in analyzing the voice of consumer primarily to address the following issues, which we may also distinguish based on the the content, sources, and temporal variation that they focus on:
1. Brand Sentiment Analysis, is concerned with measuring the sentiment expressed in the context of particular brands, products, and services, or even specific pre-defined features of a product or service. The emphasis is on volumes, and on tracking the overall aggregate positivity / negativity associated with the set of concepts one is interested in. Source selection is broad and channel based; thus one might choose to focus on say Twitter, a Facebook page, and selected blogs, as well as analyze the variation across these. Since sentiment is noisy and varies rapidly, it is also aggregated temporally; thus the time- scales of aggregate sentiment analysis are in the range of days and weeks. Social-media-based brand sentiment analysis is cheaper and faster than traditional survey-based techniques such as Nielsen market-surveys; it also reveals results sooner. Thus, sudden and significant changes in sentiment about a brand can be detected faster, such as that which took place when Tropicana changed its packaging a few years ago, which was followed by strong negative consumer sentiment. However, the jury remains out as to how often these aggregate sentiment figures bring novel insights as compared to traditional measures. The fact is they need to be time-averaged to make any sense; thus finer-grained approaches are needed to enable more real-time response, and detect emerging problems that by themselves may not change the aggregate sentiment significantly, at least at first, and that too only if not addressed in time. Listening to consumer sentiment on social platforms has recently become almost a commodity offered by a number of commercial services, such as Radian61 and others. Opinion-mining techniques for extracting sentiment from text are used in such tools. The initial insight that is most often sought through the adoption of a listening service is the ability to monitor brand perception, i.e., whether consumers at large are saying positive or negative things about one’s brand, product, or service.
2. omplaint Analysis, in contrast with brand sentiment analysis that casts its net wide, complaint analysis tries to focus on actual customers. Thus, the sources for such analysis are either direct customer feedback through call-centers or email, or when it comes to social-media, the input is carefully filtered so as to ensure the presence of indicators such as “I bought”, “my car”, etc., making it highly likely that the writer is in fact a customer, either of one’s own product or that of a competitor. Next, such complaint analysis aims to analyze the text written by customers to detect which aspects of a product or service they are having difficulty with. This requires a deeper level of natural language processing than, say, aggregate sentiment analysis: Consider the statement “I’ve been having trouble with my new [car- brand], not only did the transmission give way in the first month but there was a significant delay in getting it changed”. Clearly it is a negative statement about the car brand, and even its transmission, which basic sentiment analysis can easily discover. However, deeper text processing can further discern what exactly is wrong with the transmission, and aggregate such difficulties across a large volume of customer feedback along various dimensions. As a result, if the concept of say, transmission ‘giving way’ including its linguistic equivalents, is showing up in significant numbers, then this becomes an issue to flag to product engineering. On the other hand, the fact that the supply of spares of various types are delayed, including transmission parts etc., gets aggregated at a different level of say ‘delayed parts’, and is escalated to those responsible for after-sales services. The deeper degree of text processing required for complaint analysis requires ‘ontology-driven causal analysis’, which involves some level of parsing as well as learning, and exploiting domain ontology.
RSS Feed