About the talk
The testing pyramid is common knowledge. However, in the world of data engineering, building the whole pyramid is usually insufficient as well as impossible. It turns out that after component testing, you can't make acceptance tests. What should we do in such a case? Testing data pipelines can be seen not as just a pyramid but more like a mountain ridge with multiple peaks. In this session, we will walk through testing data pipelines from the basic unit tests to all mounts and describe why simply testing is not enough.
This talk will cover
- The usual pyramid of testing
- Parts of the pyramid applicable to testing data pipelines
- What and how we test when it comes to data pipelines
- Differences between testing traditional backends and data pipelines
About the speaker
As a Developer Advocate for Big Data at JetBrains, Pasha helps to create tools for data engineers. When he's not advocating, he writes in Kotlin, and speaks at conferences. He is also the author and maintainer of Kotlin API for Apache Spark.