Apache Spark

Code generation

A description about whole stage execution.

Interesting papers

• SparkCruise: Handsfree Computation Reuse in Spark

The idea is finding common subexpressions through logging, and selectively materialize such subquery to improve the performance. This is similar to CloudView.

References

[Mastering Spark SQL]