Professional Data Engineer Practice Test

Professional Data Engineer Practice Test practice questions, page 7 of 14, with answer review and explanations.

Question 31

You have Google Cloud Dataflow streaming pipeline running with a Google Cloud Pub/Sub subscription as the source. You need to make an update to the code that will make the new Cloud Dataflow pipeline incompatible with the current version. You do not want to lose any data when making this update. What should you do?

  1. Update the current pipeline and use the drain flag.
  2. Update the current pipeline and provide the transform mapping JSON object.
  3. Create a new pipeline that has the same Cloud Pub/Sub subscription and cancel the old pipeline.
  4. Create a new pipeline that has a new Cloud Pub/Sub subscription and cancel the old pipeline.

Question 32

Your company is running their first dynamic campaign, serving different offers by analyzing real-time data during the holiday season. The data scientists are collecting terabytes of data that rapidly grows every hour during their 30-day campaign. They are using Google Cloud Dataflow to preprocess the data and collect the feature (signals) data that is needed for the machine learning model in Google Cloud Bigtable. The team is observing suboptimal performance with reads and writes of their initial load of 10 TB of data. They want to improve this performance while minimizing cost. What should they do?

  1. Redefine the schema by evenly distributing reads and writes across the row space of the table.
  2. The performance issue should be resolved over time as the site of the BigDate cluster is increased.
  3. Redesign the schema to use a single row key to identify values that need to be updated frequently in the cluster.
  4. Redesign the schema to use row keys based on numeric IDs that increase sequentially per user viewing the offers.

Question 33

Your software uses a simple JSON format for all messages. These messages are published to Google Cloud Pub/Sub, then processed with Google Cloud Dataflow to create a real-time dashboard for the CFO. During testing, you notice that some messages are missing in the dashboard. You check the logs, and all messages are being published to Cloud Pub/Sub successfully. What should you do next?

  1. Check the dashboard application to see if it is not displaying correctly.
  2. Run a fixed dataset through the Cloud Dataflow pipeline and analyze the output.
  3. Use Google Stackdriver Monitoring on Cloud Pub/Sub to find the missing messages.
  4. Switch Cloud Dataflow to pull messages from Cloud Pub/Sub instead of Cloud Pub/Sub pushing messages to Cloud Dataflow.

Question 34

Flowlogistic Case Study - Company Overview - Flowlogistic is a leading logistics and supply chain provider. They help businesses throughout the world manage their resources and transport them to their final destination. The company has grown rapidly, expanding their offerings to include rail, truck, aircraft, and oceanic shipping. Company Background - The company started as a regional trucking company, and then expanded into other logistics market. Because they have not updated their infrastructure, managing and tracking orders and shipments has become a bottleneck. To improve operations, Flowlogistic developed proprietary technology for tracking shipments in real time at the parcel level. However, they are unable to deploy it because their technology stack, based on Apache Kafka, cannot support the processing volume. In addition, Flowlogistic wants to further analyze their orders and shipments to determine how best to deploy their resources. Solution Concept - Flowlogistic wants to implement two concepts using the cloud: ✑ Use their proprietary technology in a real-time inventory-tracking system that indicates the location of their loads ✑ Perform analytics on all their orders and shipment logs, which contain both structured and unstructured data, to determine how best to deploy resources, which markets to expand info. They also want to use predictive analytics to learn earlier when a shipment will be delayed. Existing Technical Environment - Flowlogistic architecture resides in a single data center: ✑ Databases 8 physical servers in 2 clusters - SQL Server `" user data, inventory, static data 3 physical servers - Cassandra `" metadata, tracking messages 10 Kafka servers `" tracking message aggregation and batch insert ✑ Application servers `" customer front end, middleware for order/customs 60 virtual machines across 20 physical servers - Tomcat `" Java services - Nginx `" static content - Batch servers ✑ Storage appliances - iSCSI for virtual machine (VM) hosts - Fibre Channel storage area network (FC SAN) `" SQL server storage - Network-attached storage (NAS) image storage, logs, backups ✑ 10 Apache Hadoop /Spark servers - Core Data Lake - Data analysis workloads ✑ 20 miscellaneous servers - Jenkins, monitoring, bastion hosts, Business Requirements - Build a reliable and reproducible environment with scaled panty of production. ✑ Aggregate data in a centralized Data Lake for analysis ✑ Use historical data to perform predictive analytics on future shipments ✑ Accurately track every shipment worldwide using proprietary technology ✑ Improve business agility and speed of innovation through rapid provisioning of new resources ✑ Analyze and optimize architecture for performance in the cloud ✑ Migrate fully to the cloud if all other requirements are met Technical Requirements - ✑ Handle both streaming and batch data ✑ Migrate existing Hadoop workloads ✑ Ensure architecture is scalable and elastic to meet the changing demands of the company. ✑ Use managed services whenever possible ✑ Encrypt data flight and at rest ✑ Connect a VPN between the production data center and cloud environment SEO Statement - We have grown so quickly that our inability to upgrade our infrastructure is really hampering further growth and efficiency. We are efficient at moving shipments around the world, but we are inefficient at moving data around. We need to organize our information so we can more easily understand where our customers are and what they are shipping. CTO Statement - IT has never been a priority for us, so as our data has grown, we have not invested enough in our technology. I have a good staff to manage IT, but they are so busy managing our infrastructure that I cannot get them to do the things that really matter, such as organizing our data, building the analytics, and figuring out how to implement the CFO' s tracking technology. CFO Statement - Part of our competitive advantage is that we penalize ourselves for late shipments and deliveries. Knowing where out shipments are at all times has a direct correlation to our bottom line and profitability. Additionally, I don't want to commit capital to building out a server environment. Flowlogistic wants to use Google BigQuery as their primary analysis system, but they still have Apache Hadoop and Spark workloads that they cannot move to BigQuery. Flowlogistic does not know how to store the data that is common to both workloads. What should they do?

  1. Store the common data in BigQuery as partitioned tables.
  2. Store the common data in BigQuery and expose authorized views.
  3. Store the common data encoded as Avro in Google Cloud Storage.
  4. Store he common data in the HDFS storage for a Google Cloud Dataproc cluster.

Question 35

Flowlogistic Case Study - Company Overview - Flowlogistic is a leading logistics and supply chain provider. They help businesses throughout the world manage their resources and transport them to their final destination. The company has grown rapidly, expanding their offerings to include rail, truck, aircraft, and oceanic shipping. Company Background - The company started as a regional trucking company, and then expanded into other logistics market. Because they have not updated their infrastructure, managing and tracking orders and shipments has become a bottleneck. To improve operations, Flowlogistic developed proprietary technology for tracking shipments in real time at the parcel level. However, they are unable to deploy it because their technology stack, based on Apache Kafka, cannot support the processing volume. In addition, Flowlogistic wants to further analyze their orders and shipments to determine how best to deploy their resources. Solution Concept - Flowlogistic wants to implement two concepts using the cloud: ✑ Use their proprietary technology in a real-time inventory-tracking system that indicates the location of their loads ✑ Perform analytics on all their orders and shipment logs, which contain both structured and unstructured data, to determine how best to deploy resources, which markets to expand info. They also want to use predictive analytics to learn earlier when a shipment will be delayed. Existing Technical Environment - Flowlogistic architecture resides in a single data center: ✑ Databases 8 physical servers in 2 clusters - SQL Server `" user data, inventory, static data 3 physical servers - Cassandra `" metadata, tracking messages 10 Kafka servers `" tracking message aggregation and batch insert ✑ Application servers `" customer front end, middleware for order/customs 60 virtual machines across 20 physical servers - Tomcat `" Java services - Nginx `" static content - Batch servers ✑ Storage appliances - iSCSI for virtual machine (VM) hosts - Fibre Channel storage area network (FC SAN) `" SQL server storage - Network-attached storage (NAS) image storage, logs, backups ✑ 10 Apache Hadoop /Spark servers - Core Data Lake - Data analysis workloads ✑ 20 miscellaneous servers - Jenkins, monitoring, bastion hosts, Business Requirements - ✑ Build a reliable and reproducible environment with scaled panty of production. ✑ Aggregate data in a centralized Data Lake for analysis ✑ Use historical data to perform predictive analytics on future shipments ✑ Accurately track every shipment worldwide using proprietary technology ✑ Improve business agility and speed of innovation through rapid provisioning of new resources ✑ Analyze and optimize architecture for performance in the cloud ✑ Migrate fully to the cloud if all other requirements are met Technical Requirements - ✑ Handle both streaming and batch data ✑ Migrate existing Hadoop workloads ✑ Ensure architecture is scalable and elastic to meet the changing demands of the company. ✑ Use managed services whenever possible ✑ Encrypt data flight and at rest ✑ Connect a VPN between the production data center and cloud environment SEO Statement - We have grown so quickly that our inability to upgrade our infrastructure is really hampering further growth and efficiency. We are efficient at moving shipments around the world, but we are inefficient at moving data around. We need to organize our information so we can more easily understand where our customers are and what they are shipping. CTO Statement - IT has never been a priority for us, so as our data has grown, we have not invested enough in our technology. I have a good staff to manage IT, but they are so busy managing our infrastructure that I cannot get them to do the things that really matter, such as organizing our data, building the analytics, and figuring out how to implement the CFO' s tracking technology. CFO Statement - Part of our competitive advantage is that we penalize ourselves for late shipments and deliveries. Knowing where out shipments are at all times has a direct correlation to our bottom line and profitability. Additionally, I don't want to commit capital to building out a server environment. Flowlogistic's management has determined that the current Apache Kafka servers cannot handle the data volume for their real-time inventory tracking system. You need to build a new system on Google Cloud Platform (GCP) that will feed the proprietary tracking software. The system must be able to ingest data from a variety of global sources, process and query in real-time, and store the data reliably. Which combination of GCP products should you choose?

  1. Cloud Pub/Sub, Cloud Dataflow, and Cloud Storage
  2. Cloud Pub/Sub, Cloud Dataflow, and Local SSD
  3. Cloud Pub/Sub, Cloud SQL, and Cloud Storage
  4. Cloud Load Balancing, Cloud Dataflow, and Cloud Storage