Apache Spark Interview Questions and Answers

A Spark Partition is a logical chunk of the logical division of data that is stored on a node in the cluster. Using partitions you can distribute the data among various clusters. The name itself gives the meaning that the partition is the division of tasks or anything into smaller parts. It is considered as the essential unit of parallelism in Apache Spark. Moreover, the partition in Spark will also help in reducing the memory requirements for each node. 

 An Action in Spark is the key operation of RDD. They are mainly used to access the actual and accurate data present in RDD. The key role of Action is to return the value to the driver program after running a computation on the dataset. The output of the transformation RDD operation is considered as the input of Action operation. Actions are used to return the result for the specific action performed in Spark.

Spark Core is considered as the central unit of functionality in Apache Spark. It is a key essential component of Apache Spark and it is a base unit for any kind of project you deal with. It is named as the distributed execution engine with all the functionalities. It provides a various set of functionalities like memory management, task dispatching, fault tolerance, task scheduling, in-memory computation, and many more. 

Apache Spark RDD – Resilient Distributed Dataset is formally considered as a fundamental data structure of Spark. The Apache Spark entirely revolves around the Resilient Distributed Datasets (RDDs). It is defined as the immutable collection of elements or records partitioned across the various nodes of a cluster. Using RDD the execution of various parallel operations becomes easier. There are two different ways to create Spark RDD, the first way is parallelization of existing data in the driver program and the other is through referencing a dataset in an external storage system. Spark makes use of the RDD concept to achieve efficient and faster MapReduce operations. RDD supports in-memory processing computation. Data sharing is faster in Spark RDD when compared to other networks or a disk. 

There are two main operations provided by RDD. They are:

• Transformation
• Action

The transformation operation key goal is to create a new dataset from the existing one. The transformation is a bit lazy because it only performs computation when the action requires the result to return to the driver program. The frequently used RDD transformations are as follows:

• mapPartitions(func)
• union(otherDataset)
• map(func)
• flatMap(func)

An Action in Spark is the core operation of RDD. The key role of Action is to return the value to the driver program after running a computation on the dataset. Actions are used to return the result for the specific action performed in Spark. The output of the transformation RDD operation is considered as the input of Action operation. They are mainly used to access the actual and accurate data present in RDD. The frequently used RDD actions are listed below:

• count()
• reduce(func)
• take(n)
• first()
• takeOrdered(n, [ordering])
• saveAsSequenceFile(path)(Java and Scala)
• collect()

Spark Driver program plays a key role in launching various parallel operations on the cluster. The driver in the context of Spark is also responsible for converting an application into smaller units called tasks. It is the program used to declare the actions and transformations on RDDs and also submits such requests to the master.

The built-in functions in Spark are listed below: 

• Cartesian Function
• Filter Function
• First Function
• reducedByKey Function
• Co-Group Function
• Intersection Function
• Union Function
• Count Function
• Map Function
• Distinct Function
• Take function

In Spark memory tuning there are three considerations. The considerations are as follows: 

• The memory used by the objects
• The overhead of garbage collection
• The cost of accessing those objects

Spark Parquet is the best and good looking columnar format designed and structured in a good format. The Parquet in Spark is supported by many data processing systems. When anyone who is reading the Parquet files, all the columns in the file are converted to nullable automatically for a compatibility reason. The support is provided by Spark SQL for both reading and writing Parquet files. 

The key features of Spark are as follows:

• Provides high performance for both streaming data processing and batch processing
• Easy to use
• Fast processing speed
• Enhanced support for sophisticated analytics
• Real-time stream data processing
• Flexible enough
• Provides a collection of libraries
• Hadoop Integration
• Supports multiple formats