Intel has been developing big data analytics frameworks and libraries built on Software Defined Infrastructure with open standard building blocks.  From open enterprise-ready software platforms to analytics building blocks, runtime optimizations, tools, benchmarks, and use cases, Intel® software makes big data and analytics faster, easier, and more insightful.  Examples include Apache Hadoop and Spark optimized frameworks, Intel Data Analytics Acceleration Libraries (Intel DAAL), and BigDL: Distributed Deep Learning on Apache Spark which runs over Intel Math Kernel Library for Deep Neural Networks (Intel MKL-DNN).

These Intel frameworks and libraries are being integrated with Intel FPGA acceleration options.  Customers can run unmodified applications which at run time can run on Intel Xeon®, or Intel FPGA, or other Intel platforms.  Intel is also providing FPGA acceleration frameworks with end to end orchestration, virtualization, and security.  Intel together with a partner ecosystem is offering unstructured, NoSQL, and relational data store acceleration with multi-function single Intel FPGAs, which accelerate data streams, networking, data access, and algorithms.

Traditional relational databases can benefit from significant acceleration with inline acceleration and protocol offload of networking, data streaming, and data access.   Inline accelerators include compression, filtering, and encryption.  The FPGA can also be used for memory access tasks such as cache management or memory mapped access.  Indexing/lookups and filtering run very fast as FPGA’s excel at hashing and pattern matching with their flexible datapaths.

A strong requirement is that customer's SQL applications and database schemas should run without change.  Swarm64 AS, an Intel partner, delivers a turnkey acceleration solution, Swarm64 SDA, for PostgreSQL, MariaDB, and MySQL databases. The Swarm64 SDA combines a PCIe* FPGA card with associated driver software for Linux* and a software plugin to the standard interfaces for those databases. 

Swarm64 AS has demonstrated acceleration of high velocity data and real time analytics on PostgreSQL 9.6 using an Intel Xeon E5-2695v4 with 256GB of memory and an Intel Arria® 10-based PCIe card.  Swarm64 AS achieved a more than 5X acceleration of data inserts and queries in the PostgreSQL 9.6 database accelerated by Swarm64 SDA versus a native PostgreSQL 9.6 database. Swarm64 predicts that they will achieve about 2X overall acceleration of traditional data warehousing applications and over 3X storage compression. To learn more about this solution, view this short video

Intel is developing better compression for Hadoop/Spark reduce or “shuffle” phase with an approach which completely hides the FPGA by integrated to the Intel frameworks. 

There are three additional opportunities for Spark acceleration:

  • Ingest/Kafka: Extract, Transform, Load (ETL) and filtering (partners)
  • BigDL: Deep Learning acceleration (investigation)
  • Machine Learning MLlib: ALS, other… (MCP POC)

Significant offloads are also available for NoSQL data stores.  As one example, Intel partner AlgoLogic FPGA accelerates Key Value Store with networking protocol offload, local cache management, and fast lookups.  AlgoLogic has demonstrated 3X messages per second at less than one tenth the latency.  They also demonstrated that with FPGA acceleration, that the latency is predictable, compared to software-only KVS.  This predictable latency is important when writing Service Level agreements to keep 99% latency below a certain value. 

† Tests measure performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase.  For more complete information about performance and benchmark results, visit