In-memory BI Solutions
August 25, 2010 - If you follow the business intelligence industry, you'll have been swamped with amazing claims for 'revolutionary', 'next generation' in-memory BI tools. But what is the truth behind this development?
Why all the hype?
There's one obvious reason for in-memory BI - it's extremely fast, which
is a big plus for users. The biggest performance bottleneck in typical BI
applications is slow disk or even slower database access, which is hundreds of
times slower than RAM access. Of course, disk access is not the only
bottleneck, so in-memory tools are not hundreds of times faster overall than
disk-based tools.
But that performance triggers a virtuous cycle: if an application is
intrinsically very fast by default, it doesn't need complicated data structures
to further optimize performance. Even a simple, inefficient RAM-based
application is likely to be at least an order of magnitude faster than even a
very well optimized disk-based application. That saves time and effort (which
translates into reduced consulting costs) when developing applications, and
makes it easier for in-memory applications to re-structure dynamically or
recalculate on-the-fly. This ease of development and flexibility is perhaps the
biggest advantage of in-memory BI.
However, in-memory tools, though simpler to build than disk-based tools
do need to hold data as compactly as possible. It is not enough simply to load
disk structures into memory, as these tend to store data rather inefficiently.
The best in-memory BI tools can take an order of magnitude less space for the
data than it would take in an RDBMS.
The current wave of in-memory euphoria was started by QlikTech and other smaller
vendors, but SAP, IBM, MicroStrategy and Microsoft have now joined
the bandwagon.
Why has in-memory BI suddenly become so fashionable?
In-memory applications have always been very fast, of course, but two
other more recent developments help in-memory BI: the plunging cost of memory
chips and 64-bit computers. One less obvious trend that helps in-memory BI is that
modern computers crash much less often than in the past, so there's little risk
of randomly losing all the work performed in a session.
At first glance, 64-bit computing sounds like the most important
breakthrough, as it allows access to far more RAM than the two or four GB
accessible on 32-bit systems. However, it has always been possible to access
more memory than the operating system can address, using techniques like bank
switching.
Back in the days of 16-bit computers, these techniques allowed applications
to access more than the maximum 640 KB of conventional memory. For example, in
1988, Lotus, Intel and Microsoft created a specification for an expanded memory
system (LIM-EMS) that allowed 64 KB chunks of applications data to be paged
into the reserved upper memory space - it may not sound like much today, but if
most of the conventional memory was already filled with programs, an extra 64
KB for data could double the space available for application data (such as
Lotus 1-2-3 spreadsheets).
Microsoft and Digital Research then added more features to their MS-DOS
and DR-DOS operating systems that allowed extended memory to be used beyond the
1 MB addressable by 16-bit processors, for example to hold TSR programs and DOS
components or as a high speed RAM disk.
Thus, long before 32-bit computers became established, 16-bit DOS
computers routinely used more than the supposed upper limit of 1 MB of RAM.
Exactly the same would have happened with 32-bit computers and their 4GB limit
(once thought unimaginably high), but 64-bit processors became widely available
at affordable prices early enough for these techniques not to be re-invented.
So the real breakthrough is the plunging cost of RAM, for which the BI
industry can take no credit. It is the Asian semiconductor industry's sustained
heavy investment in advanced chip foundries that has made in-memory
applications of all types practical and affordable. And that investment was not
even driven by the needs of the BI industry, but consumer demand for electronic
devices with ever more RAM. So today's in-memory BI wave is thanks to demand in
the consumer hardware market, rather than innovations in the business software
industry.