Unit 3 : Evaluating AI Models

2 MARKS QUESTIONS AND ANSWERS

1. What is the importance of model evaluation in the AI project cycle?

Ans. Model evaluation is an essential part of the model development process. It helps to find the best model that represents our data and to determine how well the chosen model will perform in the future.

2. What is meant by the "train-test split" technique?

Ans. The train-test split is a technique used to evaluate the performance of a machine learning algorithm. It involves dividing a dataset into two subsets: a training dataset for the model to learn from and a testing dataset to estimate the model's performance on new data.

3. Explain the concepts of Accuracy and Error in simple terms.

Ans. Accuracy measures the total number of correct predictions a model makes. Error refers to the difference between a model's prediction and the actual outcome, quantifying how often it makes mistakes.

4. What is a Confusion Matrix, and what are its two axes?

Ans. A Confusion Matrix is a table that presents a model's accuracy with two or more classes. The table's axes represent the actual values on the y-axis and the predicted values on the x-axis.

5. Define True Positive (TP) and True Negative (TN).

Ans. A True Positive (TP) is the outcome where the model correctly predicts the positive class. A True Negative (TN) is the outcome where the model correctly predicts the negative class.

6. Define False Positive (FP) and False Negative (FN).

Ans. A False Positive (FP) is when a model wrongly predicts a negative class as a positive class. A False Negative (FN) is when a model wrongly predicts a positive class as a negative class.

7. Why is it not recommended to use the same data for training and evaluation?

Ans. It is not recommended because the model might simply memorize the entire training set, which is a problem known as overfitting. This would lead to inaccurate performance estimation on new, unseen data.

8. What is the key goal of model evaluation in terms of accuracy and error?

Ans. The primary goal is to minimize error and maximize accuracy.

9. What is a classification problem in machine learning? Give an example.

Ans. A classification problem is one where a specific class label is the result to be predicted from the given input data. An example is a model that predicts whether an item is a vegetable or a grocery item in a supermarket.

10. What is overfitting?

Ans. Overfitting occurs when a model remembers the entire training set, leading it to always predict the correct label for any point within the training set. This results in poor performance on new data.

11. Explain the relationship between a model's performance and its accuracy.

Ans. A model's performance and accuracy are directly proportional. This means that the better the performance of the model, the more accurate its predictions are.

12. What are the two main pillars of model evaluation mentioned in the document?

Ans. The two main pillars of model evaluation are accuracy and error.

13. Why is it important to understand both error and accuracy?

Ans. Understanding both error and accuracy is crucial for effectively evaluating and improving AI models. It provides a comprehensive view of how well the model is performing and where it is making mistakes.

14. When is Accuracy not an ideal metric to use for model evaluation?

Ans. Accuracy is not ideal when there is an unequal number of observations in each class, also known as an unbalanced dataset. In such cases, other metrics are more suitable.

15. What does the F1-Score measure?

Ans. The F1-Score provides a single measure that combines both precision and recall. It is used in unbalanced datasets where it is hard to decide whether False Positives or False Negatives are more important.

16. In the context of a medical diagnosis, what is a False Negative?

Ans. A False Negative is when a model wrongly predicts a person with a disease as not having the disease. This is a critical error as the person might not get the necessary treatment.

17. What is the Abs in the context of calculating error, and why is it used?

Ans. Abs stands for absolute value. It is used to find only the magnitude of the difference between the actual and predicted values, without any negative sign.

18. Briefly explain the purpose of the train-test split.

Ans. The purpose of the train-test split is to estimate the performance of a machine learning model on new data that was not used for training.

19. What is the objective of the train-test split procedure?

Ans. The objective is to estimate the performance of a machine learning model on new data. This is how we expect to use the model in practice, by fitting it on available data and then making predictions on new examples.

20. Give one example of a situation where Precision is more important than Recall.

Ans. In the case of predicting a good day to launch a satellite, Precision is more important. Incorrectly predicting a bad weather day as a good weather day (False Positive) can be disastrous.

3 MARKS QUESTIONS AND ANSWERS

1. Explain the concept of True Positive (TP), False Positive (FP), False Negative (FN), and True Negative (TN) in the context of a confusion matrix.

Ans.

True Positive (TP): The model correctly predicted a positive outcome (e.g., a person has a disease, and the model predicted "yes").

False Positive (FP): The model wrongly predicted a negative outcome as positive (e.g., a person does not have a disease, but the model predicted "yes").

False Negative (FN): The model wrongly predicted a positive outcome as negative (e.g., a person has a disease, but the model predicted "no").

True Negative (TN): The model correctly predicted a negative outcome (e.g., a person does not have a disease, and the model predicted "no").

2. How is a Confusion Matrix a "handy presentation" of a model's accuracy?

Ans. A Confusion Matrix is a handy presentation because it provides a clear breakdown of all possible prediction outcomes. By showing the number of True Positives, True Negatives, False Positives, and False Negatives, it allows us to see not just how many correct predictions were made, but also the types of mistakes the model is making.

3. Explain the difference between Accuracy and Error in machine learning.

Ans. Accuracy and error are two sides of the same coin. Accuracy measures the total number of correct predictions a model makes, while error quantifies the number of mistakes. The goal is to maximize accuracy and minimize error. For example, if a model correctly predicts 90 out of 100 cases, its accuracy is 90%, and its error rate is 10%.

4. Describe the process of a train-test split and why it is an important step in model evaluation.

Ans. The train-test split involves dividing a dataset into a training set and a testing set. The model is trained on the training data and then evaluated on the testing data. This is important because it allows us to estimate the model's performance on new, unseen data, which is how the model will be used in practice. It helps to prevent overfitting.

5. Explain why focusing solely on accuracy might not be ideal, especially in certain real-world applications. Use an example.

Ans. Focusing only on accuracy can be misleading, especially with unbalanced datasets. For example, in a medical diagnosis, a model might have high accuracy by simply predicting "no disease" for every patient if the disease is very rare. However, this model would fail to identify any of the actual sick people. In such cases, a model with slightly lower accuracy but a strong focus on avoiding critical errors is preferable.

6. Explain the concept of Precision and when it is the most suitable metric for evaluating a model.

Ans. Precision is the ratio of correctly classified positive examples to the total number of positive predictions made by the model. It is most suitable for unbalanced datasets where minimizing False Positives (FPs) is crucial. An example is a satellite launch weather prediction model, where wrongly predicting a bad weather day as good would be disastrous.

7. Explain the concept of Recall and when it is the most suitable metric for evaluating a model.

Ans. Recall is the measure of a model's ability to correctly identify all True Positives. It is used when minimizing False Negatives (FNs) is critical. A good example is a medical diagnosis for a dangerous disease like Covid-19, where a False Negative (falsely predicting a sick person as healthy) could be life-threatening and lead to the spread of the disease.

8. What is the F1-Score, and why is it a useful metric?

Ans. The F1-Score combines both precision and recall into a single measure. It is particularly useful in situations where the dataset is unbalanced, and it's not clear whether False Positives or False Negatives are more important. A high F1-Score indicates that the model has a good balance between precision and recall.

9. Consider a spam email detection system. Which metric is more important: Precision or Recall? Justify your answer.

Ans. In a spam email detection system, Precision is more important. This is because a False Positive (classifying a legitimate email as spam) is a major problem, as a user might miss important information. While a False Negative (not detecting a spam email) is also an issue, it is considered less severe than a False Positive. Precision focuses on reducing False Positives, making it the more suitable metric in this case.

10. Explain the ethical concern mentioned in the document regarding medical diagnosis.

Ans. The ethical concern mentioned is that a model might be designed with slightly lower accuracy but a strong focus on avoiding the incorrect identification of a healthy person as sick. This is because a False Positive could lead to unnecessary and potentially harmful medical procedures. The choice of the right metric depends on the specific task and its ethical implications.

11. You are building a credit scoring model to predict if an applicant will default on a loan. Which metric would be more important, Precision or Recall? Justify.

Ans. For a credit scoring model, Recall is the more important metric. A False Negative occurs when a person who is going to default is classified as not defaulting. This is a critical business risk, as it leads to loaning money to someone who won't pay it back. The model needs to reduce False Negatives as much as possible, which is what Recall measures.

12. Calculate the Accuracy, Precision, and Recall for the following case study:

Case Study: A spam email detection system classifies 1000 emails. TP = 150, FP = 50, TN = 750, FN = 50.

Ans.

Accuracy:
(TP + TN) / (TP + FP + FN + TN)
= (150 + 750) / (150 + 50 + 50 + 750)
= 900 / 1000
= 0.9 or 90%

Precision:
TP / (TP + FP)
= 150 / (150 + 50)
= 150 / 200
= 0.75 or 75%

Recall:
TP / (TP + FN)
= 150 / (150 + 50)
= 150 / 200
= 0.75 or 75%

13. Calculate the Accuracy, Precision, and Recall for the following case study:

Case Study: A credit scoring model predicts loan default. Out of 1000 applicants, TP = 90, FP = 40, TN = 820, FN = 50.

Ans.

Accuracy:
(TP + TN) / (Total)
= (90 + 820) / (90 + 40 + 50 + 820)
= 910 / 1000
= 0.91 or 91%

Precision:
TP / (TP + FP)
= 90 / (90 + 40)
= 90 / 130
= 0.69 or 69% (approx.)

Recall:
TP / (TP + FN)
= 90 / (90 + 50)
= 90 / 140
= 0.64 or 64% (approx.)

14. Calculate the Accuracy, Precision, and Recall for the following case study:

Case Study: A fraud detection system classifies 1000 transactions. TP = 80, FP = 30, TN = 850, FN = 40.

Ans.

Accuracy:
(TP + TN) / (Total)
= (80 + 850) / (80 + 30 + 40 + 850)
= 930 / 1000
= 0.93 or 93%

Precision:
TP / (TP + FP)
= 80 / (80 + 30)
= 80 / 110
= 0.73 or 73% (approx.)

Recall:
TP / (TP + FN)
= 80 / (80 + 40)
= 80 / 120
= 0.67 or 67% (approx.)

15. Calculate the Accuracy, Precision, and Recall for the following case study:

Case Study: A medical diagnosis system classifies 1000 patients. TP = 120, FP = 20, TN = 800, FN = 60.

Ans.

Accuracy:
(TP + TN) / (Total)
= (120 + 800) / (120 + 20 + 60 + 800)
= 920 / 1000
= 0.92 or 92%

Precision:
TP / (TP + FP)
= 120 / (120 + 20)
= 120 / 140
= 0.86 or 86% (approx.)

Recall:
TP / (TP + FN)
= 120 / (120 + 60)
= 120 / 180
= 0.67 or 67% (approx.)

16. Calculate the Accuracy, Precision, and Recall for the following case study:

Case Study: An inventory management system predicts out-of-stock products. Out of 1000 products, TP = 100, FP = 50, TN = 800, FN = 50.

Ans.

Accuracy:
(TP + TN) / (Total)
= (100 + 800) / (100 + 50 + 50 + 800)
= 900 / 1000
= 0.90 or 90%

Precision:
TP / (TP + FP)
= 100 / (100 + 50)
= 100 / 150
= 0.67 or 67% (approx.)

Recall:
TP / (TP + FN)
= 100 / (100 + 50)
= 100 / 150
= 0.67 or 67% (approx.)

17. Explain the difference between Precision and Recall. Provide a use case for each.

Ans.

Precision: Precision measures the accuracy of the positive predictions, answering the question, "Of all the times the model predicted positive, how many were actually positive?". It's crucial for cases where False Positives are costly, like a satellite launch prediction.

Recall: Recall measures the model's ability to find all the actual positive cases, answering the question, "Of all the actual positive cases, how many did the model find?". It's crucial for cases where False Negatives are costly, like a medical diagnosis for a serious disease.

18. What does the term "overfitting" mean in the context of model evaluation, and why should it be avoided?

Ans. Overfitting is a phenomenon where a model learns the training data too well, to the point of memorizing it. This makes the model perform exceptionally well on the training data but poorly on any new data. It should be avoided because the objective is to create a model that generalizes well to new data, not just one that performs well on data it has already seen.

19. Calculate the F1-Score for the following case study:

Case Study: A medical diagnosis system classifies 1000 patients. TP = 120, FP = 20, TN = 800, FN = 60.

Ans.

Precision:
TP / (TP + FP)
= 120 / (120 + 20)
= 120 / 140
= 0.857

Recall:
TP / (TP + FN)
= 120 / (120 + 60)
= 120 / 180
= 0.667

F1-Score:
2 × (Precision × Recall) / (Precision + Recall)
= 2 × (0.857 × 0.667) / (0.857 + 0.667)
= 2 × 0.5716 / 1.524
= 1.1432 / 1.524
= 0.750 (approx.)

20. Explain how Precision and Recall are calculated using a Confusion Matrix.

Ans.

Precision: Precision is calculated by dividing the number of True Positives (TP) by the sum of True Positives and False Positives (TP + FP). This represents the ratio of correctly predicted positive cases out of all cases predicted as positive.

Formula:
Precision = TP / (TP + FP)

Recall: Recall is calculated by dividing the number of True Positives (TP) by the sum of True Positives and False Negatives (TP + FN). This represents the ratio of correctly predicted positive cases out of all actual positive cases.

Formula:
Recall = TP / (TP + FN)