Accelerating EMR runtime with XONAI

Just a month ago, we introduced a new type of platform that transparently accelerates data compute engines and integrates with cloud products for large-scale Big Data processing, such as AWS EMR. As announced, initial benchmarks delivered more than 2x faster Apache Spark execution on Graviton 3 machines, and 39% faster execution than EMR-accelerated Spark on Graviton 2 machines.

Today, we’re announcing seamless integration of our platform with EMR runtime, stacking acceleration in both engines and delivering 53% faster performance than EMR alone in TPC-H derived benchmarks.

As illustrated in Figure 1, the EMR runtime optimizes the selected Spark physical plan with a number of plan transformations before ultimately generating optimized Java code with the default Spark Catalyst engine. As the EMR runtime is 100% API-compatible with Spark, our platform can now correctly interface with EMR-optimized Spark, provided that it correctly inserts data conversions as needed (between row and columnar formats) in plans modified by the EMR runtime.

‍

‍

We performed a new TPC-H derived benchmark in similar conditions as the one we first announced for Graviton 2 machines, only this time comparing Spark on EMR with and without our platform. As seen in Figure 2, stacking our engine with EMR-accelerated Spark visibly outperforms the EMR runtime alone in every query. This is the expected behavior as our engine is now accelerating directly on top of EMR-optimized plans. The Q7 in particular, which was underperforming in our first benchmark, is now 30% faster.

‍

‍

As we updated the previous benchmarks only with the new EMR runtime compatibility layer, operators such as broadcast hash join and (the backported) shuffled hash join still fallback to default Spark operators, penalizing execution performance with extra data conversions. In our next performance-related blog update, we will introduce compatibility with Spark 3.2.0, complete native execution and an improved memory manager, for which we expect to significantly improve TPC-H derived benchmarking performance on the EMR platform.