Terminology clarification in Spark

Question

I have a hard time distinguishing terminologies of SparkSQL. While SparkSQL are quite flexible in terms of abstraction layers, its really difficult for beginner to navigate around those options.

1. When we say " using SparkSQL to perform .....", does it mean that we can use any API/abstraction layers such as Scala, Python, HiveQL to query? As long as the core dataframe is in spark, we should be fine?

2. Can we manipulate data in both PySpark and Scala sequentially?

For example, may I clean up the data in Scala, then perform follow up manipulation in PySpark, then go back to Scala?

3. As demonstrated in the tutorial, we can query with SQL command by using the api spark.sql("My SQL command"). does it count as SQL or SPARK?

 

Answer 1

The fact is the engine is still the same, regardless of which interface language you use. For some tasks, such as special cleaning we probably do not have SQL commands, and we have to use Scala or Python. Using Zeppelin, you can switch back and forth among languages the engine supports, however it is not a common practice. For some specific tasks, you can use pure Spark SQL or if you want to use the SQL in pyspark or scala, there are functions that can help you achieve the goal.

I believe observing more examples will help you understand when you can use what.