Accuracy is not a model evaluation plan

Start with the decision, not the score

A metric is useful only in relation to the decision a model will support. Before selecting a score, write down what a false positive and a false negative cost in the target setting. The same classifier can be acceptable for one workflow and harmful in another.

This is especially important when the classes are imbalanced. A model that predicts the majority class can achieve a high accuracy while failing to identify the cases the project was meant to find.

Use a small, complementary metric set

Accuracy is the share of predictions that are correct. Precision asks how many positive predictions were correct; recall asks how many actual positives were found. F1 combines precision and recall through their harmonic mean. These metrics answer different questions, so they should not be treated as interchangeable.

• Inspect the confusion matrix and identify the positive class.
• Report precision, recall, and F1 alongside accuracy when positive-class errors matter.
• State the split strategy and keep test data out of model selection.
• Compare against a simple baseline before celebrating a more complex model.

Thresholds are part of the model behaviour

Many classifiers output scores that are converted into class labels using a threshold. Moving that threshold changes the balance between precision and recall. If a project reports only one metric at the default threshold, readers may miss a meaningful part of the decision process.

Threshold selection should happen on validation data and reflect the use case. A threshold chosen after repeatedly inspecting the test set can make the reported test result optimistic.

Make the evidence reproducible

A credible evaluation report should identify the dataset, target label, preprocessing, train/validation/test strategy, baseline, selected metric, and limitations. Where experiments are repeated, record configuration and results so comparisons do not depend on memory or notebook execution order.

For healthcare, finance, hiring, or other high-impact contexts, predictive performance alone is not evidence that a model is safe or appropriate to deploy. Data representativeness, calibration, subgroup behaviour, monitoring, and human oversight need separate consideration.

A compact review checklist

Before publishing a result, ask: does the metric match the decision; is the baseline credible; was the test set held out; are the positive class and threshold explicit; and are the limits of the data clear? If any answer is missing, the next step is not a bigger model. It is better evidence.

References and further reading

These sources informed the technical framing of this note. Consult the original documentation for full definitions and context.