Certified Associate Developer for Apache Spark Practice Test

Certified Associate Developer for Apache Spark Practice Test practice questions, page 10 of 14, with answer review and explanations.

Question 46

Which of the following operations can be used to return a new DataFrame from DataFrame storesDF without inducing a shuffle?

  1. storesDF.intersect()
  2. storesDF.repartition(1)
  3. storesDF.union()
  4. storesDF.coalesce(1)
  5. storesDF.rdd.getNumPartitions()

Question 47

The code block shown below contains an error. The code block is intended to return a new 12-partition DataFrame from the 8-partition DataFrame storesDF by inducing a shuffle. Identify the error. Code block: storesDF.coalesce(12)

  1. The coalesce() operation cannot guarantee the number of target partitions – the repartition() operation should be used instead.
  2. The coalesce() operation does not induce a shuffle and cannot increase the number of partitions – the repartition() operation should be used instead.
  3. The coalesce() operation will only work if the DataFrame has been cached to memory – the repartition() operation should be used instead.
  4. The coalesce() operation requires a column by which to partition rather than a number of partitions – the repartition() operation should be used instead.
  5. The number of resulting partitions, 12, is not achievable for an 8-partition DataFrame.

Question 48

Which of the following Spark properties is used to configure whether DataFrame partitions that do not meet a minimum size threshold are automatically coalesced into larger partitions during a shuffle?

  1. spark.sql.shuffle.partitions
  2. spark.sql.autoBroadcastJoinThreshold
  3. spark.sql.adaptive.skewJoin.enabled
  4. spark.sql.inMemoryColumnarStorage.batchSize
  5. spark.sql.adaptive.coalescePartitions.enabled

Question 49

The code block shown below contains an error. The code block is intended to return a DataFrame containing a column openDateString, a string representation of Java’s SimpleDateFormat. Identify the error. Note that column openDate is of type integer and represents a date in the UNIX epoch format – the number of seconds since midnight on January 1st, 1970. An example of Java’s SimpleDateFormat is "Sunday, Dec 4, 2008 1:05 PM". A sample of storesDF is displayed below: Code block: storesDF.withColumn("openDateString", from_unixtime(col("openDate"), "EEE, MMM d, yyyy h:mm a", TimestampType()))

  1. The from_unixtime() operation only accepts two parameters – the TimestampTime() arguments not necessary.
  2. The from_unixtime() operation only works if column openDate is of type long rather than integer – column openDate must first be converted.
  3. The second argument to from_unixtime() is not correct – it should be a variant of TimestampType() rather than a string.
  4. The from_unixtime() operation automatically places the input column in java’s SimpleDateFormat – there is no need for a second or third argument.
  5. The column openDate must first be converted to a timestamp, and then the Date() function can be used to reformat to java’s SimpleDateFormat.

Question 50

Which of the following code blocks returns a DataFrame containing a column dayOfYear, an integer representation of the day of the year from column openDate from DataFrame storesDF? Note that column openDate is of type integer and represents a date in the UNIX epoch format – the number of seconds since midnight on January 1st, 1970. A sample of storesDF is displayed below:

  1. (storesDF.withColumn("openTimestamp", col("openDate").cast("Timestamp")) . withColumn("dayOfYear", dayofyear(col("openTimestamp"))))
  2. storesDF.withColumn("dayOfYear", get dayofyear(col("openDate")))
  3. storesDF.withColumn("dayOfYear", dayofyear(col("openDate")))
  4. (storesDF.withColumn("openDateFormat", col("openDate").cast("Date")) . withColumn("dayOfYear", dayofyear(col("openDateFormat"))))
  5. storesDF.withColumn("dayOfYear", substr(col("openDate"), 4, 6))