“On
two occasions I have been asked,… if you put the wrong figures into the machine,
will the right answers come out. I am not rightly able to apprehend the kind of
confusion of ideas that could provoke such a question”
Charles
Babbage. 1864. Passages from the Life of a Philosopher (chap. 5).
About 9 months ago, ~ 4½ generations in data science evolution,
IDC released a report from their FutureScape Program titled: Big Data &
Analytics 2015 Predictions. It’s still 2015, & I have been doing a good
amount of work with healthcare organizations deploying & readying these
groups to use contemporary (Hadoop-based) analytics in their strategic decision
making, so,… I thought I’d look back at these predictions & react to them
from my current experience.
4½ generations old, you say… The good news about this is that
actual field deployment & use of “big data” analytics has not been growing as
fast as the amount of material written in general (& the hype) about this
topic[1].
First, deployment & use in any productive way requires some amount of
organizational change & alignment & that slows down deployment. Second,
as I have written about previously, acquisition & deployment of a
technology is not the same as its adoption[2].
This is especially true in this case as adoption requires: 1) the deployment of
additional different infrastructure & user-facing technologies &, 2)
some (not small) amount of change in the way an organization thinks about data,
the use of information & decision-making. Finally, most organizations are
conservative when it comes to this level of change, so adoption &
productive use requires management commitment & a senior champion to keep
things moving until leverage can be demonstrated. Given all this, I think we
are 2-4 years from the adoption & productive use of big data analytics
being in general use in well-resourced organizations & 5-8 years for
everyone else.
The first thing to do if I’m going to comment on IDC’s
predictions is to define what big data is. There are many, many attempts to do
this on the net, but suffice to say that there is no known (at least by me)
consensus as to what the term means. Quite a number of definitions are couched
in terms of volume of data, & most of these use ≥ 1PB as the boundary for big data. I think it is more
nuanced than that; in fact I think that at least the following considerations
are relevant:
·
Volume - ≥
1PB is as good a number as any, but much smaller amounts of data can be
productively analyzed, see material below on data volume & data quality
·
Storage technology – Currently relational-based
data warehousing is the primary enterprise systems method for the storage of
large amounts of data. This technology has definite limitations in terms of
volume of data & performance that will be covered later. More contemporary,
& effective storage technologies include: NoSQL database, graph database
& massive, parallel distributed file systems e.g. Hadoop Distributed File System (HDFS) or sparse array
technologies e.g. Google Cloud Big
Table. These last two are most associated with big data storage.
·
Variety – The variety of the data stored in a
system is also an issue. Data warehouses require a standardized data model that
all data is normalized against. Data types that cannot be normalized are not
stored in native form. Normalization & transformation is a complicated
& very time-consuming process, as is updating the model if/when data requirements
change. HDFS & Big Table like systems do not require a standardized data
model or this type of normalization. An almost unlimited number of types of
data can be stored & utilized in these systems making them much more
aligned with real-world needs.
·
Analysis technology – Many systems, regardless
of the volume of data, provide SQL based query as well as conventional BI as
the primary analysis technology. Even using Turing-complete versions of SQL (e.g. T-SQL) limits the types of queries
& analysis that can be done, though all HDFS based systems allow some form
of SQL-based query. There certainly are many more SQL programmers in the world
than data scientists who model & query in other systems/languages. Yarn
(MapReduce2) is often used for analysis with HDFS & Big Table based systems
although the most effective analysis is provided by model development in a
language such as R, Pig, Python, etc. operating against HDFA/Yarn.
·
Search - Storage of substantial amounts of
information requires a search function that is integrated & well aligned with
storage & analysis functions. Conventional search techniques used in
relational-based data warehouses tend to lower performance as data volume
increases. Applications such as Apache Solr, elasticsearch & the search
associated with HBase, Hive etc. are designed to work with massive data storage
facilities.
Prediction is hard, we are told. The great Yogi Berra is
reported to have opined that, “Prediction is hard, especially when it’s about
the future”. One of the best ways to ensure that your predictions are accurate
is to make them about the present. As a technology futurist, I’ve been guilty
of this myself. Many of the IDC predictions are just this. They are not wrong…
they may be accurate & prescient statements about the state of the
technology – they just are not predictions.
So what are the IDC predictions? I’ll list them here with
associated comments:
1.
Visual data discovery
tools will be growing 2.5x faster than rest of the BI market. By 2018 investing
in this enabler of end user self-service will become a requirement for all
enterprises.
I think
this is essentially correct & already happening faster than this prediction.
Many organizations already use visual front-ends for their business
intelligence (BI) analysis & so their expectations are set by current
practice. Unfortunately, this often means that it is difficult to look beyond
current practice to find unique & productive ways of using tools that are
already being used in another context. IDC points out that the adoption of
visualization tools is driven by a demand for “self-service” & BYOT (tools)
in analysis. While this might be true, a more compelling motivation is that
often the results of complex modeling, or even complex statistical analysis,
are difficult to interpret if one is not a data scientist. Tools such as Qlik
& Tableau allow for quick summarization of certain types of results in an
easily understandable form. One of the downsides of the use of these tools is
that often results are complex or more nuanced than can be expressed in
visualizations. I have found that these tools are best used to summarize
results for executive review.
2.
Over the next 5 years
spending on cloud-based BDA (big data & analytics) solutions will grow 3x
faster than spending for on-premise solutions. Hybrid on/off premise
deployments will become a requirement.
Also
essentially correct, but what it says about big data & analytics is that
the majority of systems used for this effort will be cloud-hosted in some way.
This both reduces cost, as opposed to on-premise hosting, & reduces risk as
cloud services take on the maintenance & upgrade efforts needed to make the
analytic effort work.
There is a
different way of looking at this, though, one that I have to deal with
currently on several data projects. The problem that this solution tries to
address is the provision of (perceived) adequate levels of data privacy &
security. A number of companies that I’m working with have specified that no
data is to leave their security perimeter & similarly, no processing
(analysis) is to be done off-site. This was fine when companies had 1 PB (1x109) of data
but gets difficult to impossible at 500 PBs or, say, 5 TBs (5x1012). At least one of these companies has a respectable amount of
data (35-40 PBs). This company has chosen a commercial private cloud deployment
for BDA (from Rackspace) that allows for self-service provisioning of servers,
elastic scalability, multitenancy & many of the advantages of cloud-based
deployments in what is essentially a private environment. Luckily, they have a
well-resourced & capable IT group as this is not an easy deployment, even
with a commercial vendor. The most popular current private cloud deployment is
an open source package from OpenStack. There are many (many) horror stories
related to these deployments (not just OpenStack, but any of the open source
vendors). The technical & deployment issues with open source private stack
have been written about at least since mid-2013[3].
Of course, as Mr Asay & many others have pointed out, the real problem is
the contradiction of developing & deploying a cloud infrastructure for
private use. I actually think there is a place for this specific solution, but
I agree that it is much more likely to be used as a hybrid solution where data
& proprietary results may be stored privately, but processing may be done
publically, so long as results & data are stored privately. Of course, the
company I’m working with that has 40 PBs of data wanted everything on premise,
although they did the deployment at a remote data center that they lease space
at, so everything is relative…
3.
Shortage of skilled
staff will persist. In the U.S. alone there will be 181K deep analytics roles
in 2018 and 5x that many positions requiring related skills in data management
and interpretation.
I don’t
know how IDC derived these numbers, but they are probably as good as any. The
thing that I think has to be addressed is how do we promote & advance
analytics without having to wait for 181,000 doctoral level data scientists to
be trained. It’s Q4 2015, can this many people be trained by 2018? Probably
not… This means that we have to “make due” with what we have. In part we are
already doing this by using SQL as a means of querying HDFS data, but we need
to go beyond this. SQL is (kinda) OK as a database query language[4],
but it was never intended, despite the provision for stored procedures[5],
as a modeling & functional execution language. Data analysts need to begin
to learn to use additional languages to model & query the data in massive
stores.
As of
today, the primary languages in use for this are: R, Python, Java, Scala, Pig
Latin & various additions to the Hadoop stack such as Hive. R is the most
used analytic language at present. It is excellent for complex statistical
models & analysis & has a very rich ecosystem of added functions &
libraries. It is probably just on the cusp of being superseded, primarily
because it has some size limitations with respect to how much data it can deal
with. Python is easier to learn & use than R, but it still has some size
limitations & is not highly performant at scale. Java, of course, has the
advantage of a very large programmer base, but it is not very good at
statistical analysis (complex functions have to be written rather than called
from libraries). It does have good options for the display of statistical
results. Scala is Java-based & is mainly used today to build machine-learning
algorithms. Pig Latin is run on the Pig platform (developed at Yahoo Research
& now under Apache license) & is an abstract idiom (notation) of Java
that allows high-level programming of MapReduce jobs for analysis.
Probably
the easiest path here is to start with what you know, SQL moving to Java moving
to something like Pig to be able to write MapReduce jobs directly. Also,
keeping up with what is being put into open source in this area is important as
many languages & visual palettes that allow direct MapReduce programming
will be developed & released over the next 2-3 years.
4.
By 2017 unified data
platform architecture will become the foundation of BDA strategy. The
unification will occur across information management, analysis, and search
technology.
This has
already happened as evidenced by the offerings of Cloudera, Hortonworks, etc. I
expect that offerings of this type will continue to be more & more
integrated so that ultra-large scale information management analysis &
search will seem almost seamless. This will also apply to offerings based on
other (Non-Hadoop) platforms such as Spark.
5.
Growth in applications
incorporating advanced and predictive analytics, including machine learning,
will accelerate in 2015. These apps will grow 65% faster than apps without
predictive functionality.
This is
also essentially already happening. Very few people actually understand the
internals of machine learning, but more & more businesses are basing
business models & products on it. I think that there are two major paths
here. The first is that products in the form of applications & add-on
modules will become very important so that machine learning in some form can be
integrated into many different types of business processes. Second, these machine-learning
modules will also be integrated into the analytic stacks mentioned above…
Currently there are several such modules associated with both Hadoop (Apache
Mahout, Wabbit, Cloudera Oryx, oxdata H2O, MLLib) & Spark (MLLib, Cloudera Oryx). Most of these are trend
analysis &/or predictive algorithms. Actual machine learning (supervised or
unsupervised) integrated with analytics stacks is still a little ways off (2-3
years).
6.
70% of large organizations already purchase external data and
100% will do so by 2019. In parallel more organizations will begin to monetize
their data by selling them or providing Value Added Content.
Again, this is not a real prediction. If
70% of organizations do this today, it will be considerably before 2019 that
close to 100% of them do it. The real (IMHO) question is what kinds of data are
being used & for what purpose(s)? Most of the BDA projects that I have seen
fall into two categories: 1) optimization of operational processes &
decisions, & 2) optimization of specific knowledge-based processes &
decisions such as medical diagnosis or developing predictive trends in interest
rate movements. In the first case mostly internal data is used except for
operational benchmarking. This requires external data for comparison &
trend development. In the second case, substantial amounts of data exist
outside of a typical organization that will enhance the optimization of
knowledge-based processes. In healthcare, for instance, this might include
clinical data from partners or State & Federal sources, population health
data from partners or State & Federal sources, registry data on
immunization, best practice data from public & private sources & many
more. Most of this will require acquiring data from external sources, & in
turn, as IDC has pointed out, this will provide substantial opportunities for
organizations to monetize their data & for intermediaries to aggregate
& “sell” data.
7.
Adoption of technology to continuously analyze streams of
events will accelerate in 2015 as it is applied to IoT analytics – which is
expected to grow at a 5-year CAGR of 30%.
The main driver for this, at least
commercially, will be the internet of things (IoT), so depending on how quickly
you believe the IoT will be developed, deployed & adopted at scale, this
rate is either wildly too low or wildly too high. I currently do not see a huge
push commercially for the IoT, so I think this rate is too high. In 5 years, I
think we’ll still be looking at early adoption, especially in at-home use,
except in some specific areas. These include (but, as we futurists say, are not
limited to):
·
Healthcare – remote monitoring both in facility & at home
will increase the number of sensors that are reporting on an individual’s
health status.
·
Transportation – There are separate areas: autonomous driving
will increase the data flow as information is analyzed in both real-time &
asynchronously. This will also be true for larger scale data streams for things
like traffic control.
·
Process Manufacturing – This has already happened to a large
extent. Industries such as chemical production have used continuous monitoring
& analysis of data for years.
·
Other Manufacturing – More & more discrete manufacturing
processes are designed with this type of sensing & monitoring
8.
Decision management platforms will expand at a CAGR of 60%
through 2019 in response to the need for greater consistency in decision making
and decision making process knowledge retention.
This one I am not so sure about. As above,
I think this will vary wildly in different segments. For generalized business
& strategy decisions, I think this will continue to be a hard sell. The
technology exists now to greatly facilitate & enhance planning &
decision-making processes, but there are real social & cultural impediments
to adoption. In my experience these are of two types: 1) unfamiliarity with the
concepts & possibilities of big data analysis, & 2) organizational
& individual homeostasis that translates into resistance to change, even in
the face of real need for change. I started this essay by stating the
prediction is hard… I believe that change is even harder. People can be
educated about the concepts & function of new technology & process,
they may even come to understand the technology & its possible uses &
advantages, but if they are socially &/or culturally biased against change,
even if covertly, adoption of the new technology is very difficult. I believe
that general adoption of decision making platforms, or any other manifestation
of big data & analytics, will require some early adopters to have
unqualified successes that are obvious & evident. This may happen in
specific segments over the next 3 or so years, at which point many more
individuals & organizations, in that segment, will be more amenable to
adoption. This pattern of adoption by segment (healthcare, financial services
etc.) will ensure that general adoption beyond early adopters & risk takers
will be delayed, probably in the 5-8 year range.
9.
Rich media (video, audio, image) analytics will at least
triple in 2015 and emerge as the key driver for BDA technology investment.
I’m also not so sure about this one. I
cannot see this tripling by the end of 2015, as the investment made in general
in BDA does not support any segment of it tripling this year. I also cannot see
this being the primary driver for BDA investment anytime soon. Most companies
that are investing at this point are not primarily in the business of media
analytics, at least not in the segments I have a view into. This may be true in
the media industry, but I do not yet see a business problem &/or business
model that would lead to this rate of adoption.
The other issue here is that analysis of
this type of data is still primarily of its metadata. Analysis & more
detailed analytics of the internals (actual content) of media data types are
still research topics. It will take another 3-5 years before we have reliable
& accurate algorithms to do this type of analysis. Of course, I could be
wrong,… but…
10. By 2018
half of all consumers will interact with services based on cognitive computing
on a regular basis.
Well, this depends on what your definition of “cognitive
computing” is. TechTarget defines it as “…simulation of human thought processes
in a computerized model. Cognitive
computing involves self-learning
systems that use data mining, pattern
recognition and natural language
processing to mimic the way the human brain works.[6]”
The definition goes to say to say that the main function in these
systems is machine learning. OK, I’ve
been working in this area since the late 1970s & I am not convinced that
such systems exist today or that they will, in fact, exist in 2018. Systems
that use many of these capabilities certainly exist today. Some of those
systems are even productive & interesting in specific ways. If this
prediction is meant to mean that a large number of people will interact with
systems that have used a supervised random tree algorithm to produce an
optimized prospect list for an insurance product, then that is true today. If
this prediction is meant to mean that a large number of people will interact
with a system that has a general capability for interacting with &
responding to a person with some underspecified request, I think that it is
very unlikely in this timeframe. I have written much more extensively on this
in a previous blog post.[7]
So where are we? I think many of IDC’s predictions were, in
fact peridictions[8].
The two most interesting one (IMHO) were number 2 regarding cloud-based BDA
solutions & number 3 regarding lack of knowledgeable people to do analysis.
There are many pluses & minuses to cloud deployments, especially with
respect to security & privacy requirements. The potential impediments in
these deployments will be dealt with over the next few years by cloud vendors
& the use of hybrid clouds will, I believe, be the dominant deployment
model 5 years from now. Skilled people are another matter. If there really is a
need for close to 200K data scientists of various levels in the next 5 years,
that need cannot be met. We’ll have to proceed using what we already know &
what can be learned during this period, which includes: use of integrated
platforms that unify data storage, analytic design, execution of analysis &
visualization of results. We’ll also have to start building our analytic models
with what expertise is available… SQL moving to more suitable languages &
modeling capabilities. The one thing that we do know about big data &
analytics is that it will become increasingly important – initially more as a
way of determining what questions to ask & eventually (within 5-8 years) as
a way of developing & testing strategy[9].
Another thing we know is that this type of analytics, or any type, will not
provide actual answers for our strategic questions. That we’ll still have to do
ourselves, for the foreseeable future, until we really develop & can
interact with cognitive computing.
[1] Big Data does not even
appear on the Gartner 2015 Hype Cycle, although it was entering the “Trough of
Disillusionment” on the 2014 chart.
[2] The Coevolution of
Organizations & Technology. 1994. MIT/LFM Working Papers., A Framework
& Model for Technology Adoption in Healthcare Organizations. 2006.
PostTechnical Strategist.
[3] c.f.
Matt Asay writing for TechRepublic,
http://www.techrepublic.com/article/private-clouds-very-public-failure/
[4] The author was a member,
representing the Digital Equipment Corporation, of the ANSI X3H2 Standards
Committee that standardized SQL the first time in 1986.
[5] That I opposed during the
initial standardization & am still not a fan of today… Stored procedures
are a great way to make the function of an analysis opaque & not amenable
to modification or debugging.
[7] Turing Tests, Search &
Current AI. http://posttechnical.blogspot.com/2015/09/turing-tests-search-current-ai.html
[8] statements about the present
[9] see my blog post: Design Thinking
as Work Process & Strategy.
http://posttechnical.blogspot.com/2015/09/design-thinking-as-work-process-strategy.html
No comments:
Post a Comment