COURSE 3: BUILD AND OPERATE MACHINE LEARNING SOLUTIONS WITH AZURE

Module 6: Monitor Machine Learning Deployments

MICROSOFT AZURE DATA SCIENTIST ASSOCIATE (DP-100) PROFESSIONAL CERTIFICATE

Complete Coursera Study Guide

Enroll in Coursera Microsoft Azure Data Scientist Associate (DP-100)

Last updated:

TABLE OF CONTENT

INTRODUCTION – Monitor Machine Learning Deployments

Machine learning models can inadvertently include biases that lead to unfair outcomes. In this module, you will learn how to use Fairlearn and Azure Machine Learning to detect and mitigate such biases in your models. Additionally, you will discover how to use telemetry to monitor how a machine learning model is utilized once deployed in production. Finally, you will learn to monitor data drift to ensure your model continues to make accurate predictions over time.

Learning Objectives

  • Evaluate machine learning models for fairness.
  • Mitigate predictive disparity in a machine learning model.
  • Enable Application Insights monitoring for an Azure Machine Learning web service.
  • Capture and view model telemetry.
  • Create a data drift monitor.
  • Schedule data drift monitoring.
  • View data drift monitoring results.

PRACTICE QUIZ: KNOWLEDGE CHECK 1

1. In cases where certain groups are overrepresented or data is skewed, which potential cause of disparity applies here?

  • Societal bias
  • Data imbalance (CORRECT)
  • Indirect correlation

Correct: This happens when some groups can be overrepresented in the training data, or the data may be skewed so that cases within a specific group aren’t representative of the overall population.

2. When attempting to mitigate bias, if you want to differentiate features that are directly predictive from features that encapsulate complex or nuanced relationships, which strategy should you apply?

  • Extensive feature selection and engineering analysis. (CORRECT)
  • Evaluate models for disparity based on significant features.
  • Balance training and validation data

Correct: You can perform extensive feature selection and engineering analysis to make sure you fully explore the interconnected correlations in your data to try to differentiate features that are directly predictive from features that encapsulate more complex, nuanced relationships.

3. True or false?

Applying a trade-off overall predictive performance between sensitive feature groups means that a model that is 97.5% accurate with similar performance across all groups can be more desirable than a model that is 99.9% accurate but has a bias against a group.

  • True (CORRECT)
  • False.

Correct: The goal here is to try to minimize the discrimination against groups or cases with sensitive features by trading off a margin of accuracy.

4. Which mitigation algorithm is a post-processing technique that applies a constraint to an existing classifier and transforms the prediction as appropriate?

  • Grid Search
  • Threshold Optimizer (CORRECT)
  • Exponentiated Gradient 

Correct: This is the mitigation algorithm described.

5. Which mitigation algorithms support binary classification and regression models? Select all that apply.

  • Threshold Optimizer
  • Grid Search (CORRECT)
  • Exponentiated Gradient (CORRECT)

Correct: This algorithm supports both binary classification and regression models.

Correct: This algorithm supports both binary classification and regression models.

PRACTICE QUIZ: KNOWLEDGE CHECK 2

1. True or false?

Application Insights can also be used to monitor applications that are not running in Azure.

  • True (CORRECT)
  • False

Correct: You can use Application Insights to monitor telemetry from many kinds of application, including applications that are not running in Azure.

2. What method of the Workspace object can you use in the SDK to determine the Application Insights associated resource?

  • get_details() (CORRECT)
  • get_overview()
  • get_attachments ()
  • get_associations()

Correct: This method will retrieve a detailed overview of your Workspace object, including the associated resources.

3. In order to capture telemetry data for Application Insights, what statement can you use to write values to the standard output log in the scoring script of your service?

  • print_output
  • print (CORRECT)
  • write
  • write_output

Correct: This statement will write values to the output log of a service.

4. From the Azure Portal, which blade of the Application Insights will allow you to query log data?

  • Smart Detection
  • Log Analytics (CORRECT)
  • Metrics
  • Activity logs

Correct: To analyze captured log data, you can use the Log Analytics query interface for Application Insights in the Azure portal.

5. What query language is used to query log data in App Insights?

  • Standard Query Language (SQL)
  • Concept-Oriented Query Language (COQL)
  • Object Query Language (OQL)
  • Kusto Query Language (KQL) (CORRECT)

Correct: This interface supports a SQL-like query syntax called Kusto Query Language that you can use to extract fields from logged data, including custom dimensions created by your Azure Machine Learning service.

PRACTICE QUIZ: KNOWLEDGE CHECK 3

1. What is Azure Machine Learning using to monitor data drift?

  • Datasets (CORRECT)
  • Databases
  • Datastores

Correct: Azure Machine Learning supports data drift monitoring through the use of datasets. You can capture new feature data in a dataset and compare it to the dataset with which the model was trained.

2. Which are the two dataset types you need to register to monitor data drift? Select all that apply.

  • Primary dataset
  • Scope dataset
  • Baseline dataset (CORRECT)
  • Target dataset (CORRECT)

Correct: This is usually the original training data.

Correct: This will be compared to the baseline based on time intervals. This dataset requires a column for each feature you want to compare, and a timestamp column so the rate of data drift can be measured.

3. After creating the registered datasets, what can you define to detect data drift and trigger alerts?

  • Dataset logger
  • Dataset metrics
  • Dataset monitor (CORRECT)
  • Dataset analyzer

Correct: You can define a dataset monitor to detect data drift and trigger alerts if the rate of drift exceeds a specified threshold.

4. After creating the dataset monitor, what function can you use to compare the baseline dataset to the target dataset?

  • Backload
  • Backtrack
  • Backflow
  • Backfill (CORRECT)

Correct: You can use the backfill function to immediately compare the baseline dataset to existing data in the target dataset.

5. What are the timeframes that can be configured to run a data drift monitor schedule? Choose all that apply.

  • Hourly
  • Weekly (CORRECT)
  • Monthly (CORRECT)
  • Daily (CORRECT)

Correct: You can define a schedule to run every week.

Correct: You can define a schedule to run every month.

Correct: You can define a schedule to run every day.

QUIZ: TEST PREP

1. Which parity constraint can be used with any of the mitigation algorithms to minimize disparity in the selection rate across sensitive feature groups?

  • Equalized odds
  • Demographic parity (CORRECT)
  • Error rate parity
  • Bounded group loss

Correct: This is the parity constraint described. For example, in a binary classification scenario, this constraint tries to ensure that an equal number of positive predictions are made in each group.

2. Which parity constraint can be used with any of the mitigation algorithms to minimize disparity in combined true positive rate and false_positive_rate across sensitive feature groups?

  • Error rate parity
  • False-positive rate parity
  • True positive rate parity
  • Equalized odds (CORRECT)

Correct: This is the parity constraint described. For example, in a binary classification scenario, this constraint tries to ensure that each group contains a comparable ratio of true positive and false-positive predictions.

3. You are training a binary classification model to determine who should be targeted in a marketing campaign. 

How can you assess if the model is fair and will not discriminate based on ethnicity?

  • Remove the ethnicity feature from the training dataset.
  • Evaluate each trained model with a validation dataset, and use the model with the highest accuracy score. An accurate model is inherently fair.
  • Compare disparity between selection rates and performance metrics across ethnicities. (CORRECT)

Correct: By using ethnicity as a sensitive field, and comparing disparity between selection rates and performance metrics for each ethnicity value, you can evaluate the fairness of the model.

4. When deploying a new real-time service, Application Insights can be enabled in the deployment configuration for the service.

How would you code that using the SDK?

  • dep_config = AciWebservice.deploy_configuration(cpu_cores = 1,
  • memory_gb = 1,
  • appinsights=True)
  • dep_config = AciWebservice.deploy_configuration(cpu_cores = 1, (CORRECT)
  • memory_gb = 1,
  • enable_app_insights=True)
  • dep_config = AciWebservice.deploy_configuration(cpu_cores = 1,
  • memory_gb = 1,
  • app_insights(True))
  • dep_config = AciWebservice.deploy_configuration(cpu_cores = 1,
  • memory_gb = 1,
  • app_insights=True)

Correct: This is the correct code.

5. For web services that have already been deployed, you can update them and enable the Application Insight using the Azure ML SDK.

How would you code that?

  • service = ws.webservices(‘my-svc’)
  • service.create(enable_app_insights=True)
  • service = ws.webservices[‘my-svc’] (CORRECT)
  • service.update(enable_app_insights=True)
  • service = ws.webservices[‘my-svc’]
  • service.new (enable_app_insights=True)
  • service = ws.webservices[‘my-svc’]
  • service.modify (enable_app_insights=True)

Correct: This is the correct code.

6. You want to back fill a dataset monitor based on monthly changes in data for the previous 3 months. How would you code that in the SDK?

  • import datetime as dt (CORRECT)
  • backfill = monitor.backfill( dt.datetime.now() – dt.timedelta(months=3), dt.datetime.now())
  • import datetime as dt
  • backfill = monitor.backfill( dt.datetime.now(), dt.timedelta(months=3), dt.datetime.now())
  • import datetime as dt
  • backfill = monitor_backfill( dt.datetime.now() – dt.timedelta(months=3), dt.datetime.now())
  • import datetime as dt
  • backfill = monitor_backfill( dt.datetime.now(), dt.timedelta(months=3), dt.datetime.now())

Correct: This is the correct code.

7. You want to schedule a data drift monitor to run every day and send an alert if the drift magnitude is greater than 0.2. How would you code that in Python?

  • alert_email = AlertConfiguration(‘data_scientists@contoso.com’) (CORRECT)
  • monitor = DataDriftDetector.create_from_datasets(ws, ‘dataset-drift-detector’, 
  • baseline_data_set, target_data_set,
  •  compute_target=cpu_cluster,
  •  frequency=’Day’, latency=2,
  •  drift_threshold=.2,
  •  alert_configuration=alert_email)
  • alert_email = AlertConfiguration(‘data_scientists@contoso.com’)
  • monitor = DataDriftDetector.create_from_datasets(ws, ‘dataset-drift-detector’, 
  •  baseline_data_set, target_data_set,
  •  compute_target=cpu_cluster,
  •  frequency=’Week’, latency=2,
  •  drift_threshold=.2,
  •  alert_configuration=alert_email)
  • alert_email = AlertConfiguration(‘data_scientists@contoso.com’)
  • monitor = DataDriftDetector.create_from_datasets(ws, ‘dataset-drift-detector’, 
  •  baseline_data_set, target_data_set,
  •  compute_target=cpu_cluster,
  •  frequency=’Day’, latency=2,
  •  drift_threshold=.4)

Correct: This is the correct code for the task.

8. What happens when some groups are overrepresented in training data?

  • Data imbalance (CORRECT)
  • Indirect correlation
  • Societal biases

Correct: Data imbalance can occur when some groups may be overrepresented in the training data, so that cases within a specific group aren’t representative of the overall population.

9. True of false. 

Fairlearn includes a MetricFrame function that enables you to create a dataframe of multiple metrics by the group.

  • True (CORRECT)
  • False

Correct: Fairlearn includes a MetricFrame function that enables you to create a dataframe of multiple metrics by the group.

10. Which of the following Fairlearn constraints would you choose to ensure that the error for each sensitive feature group does not deviate from the overall error rate by more than a specified amount?

  • Demographic parity
  • False-positive rate parity
  • Error rate parity (CORRECT)

Correct: Choose the Error rate parity constraint to ensure that the error for each sensitive feature group does not deviate from the overall error rate by more than a specified amount.

11. Which code method can you use to determine the Application Insights resource associated with your workspace?

  • get_details() (CORRECT)
  • get_insights()
  • show_details()

Correct: You can use the get_details() method of a Workspace object to determine the Application Insights resource associated with your workspace.

12. Which types of datasets do you need to register to monitor data drift?

  • A target dataset only
  • A baseline dataset only 
  • A baseline dataset and a target dataset (CORRECT)

Correct: To monitor data drift using registered datasets, you need to register two datasets, A baseline dataset, and a target dataset.

CONCLUSION – Monitor Machine Learning Deployments

By mastering the use of Fairlearn and Azure Machine Learning to detect and mitigate bias, you will ensure your models produce fair outcomes. Understanding how to use telemetry will enable you to monitor model usage in production effectively. Additionally, learning to track data drift will help maintain the accuracy of your model’s predictions over time. These skills will enable you to create more reliable, fair, and accurate machine learning solutions.

Previous Module
Weekly Breakdown

Quiztudy Top Courses

Popular in Coursera

Liking our content? Then, don’t forget to ad us to your BOOKMARKS so you can find us easily!

Expanded Thumbnail Promo1

Mood Zone for Studying & Relaxing