What is the 'No-Free-Lunch Theorem' in ML?

What is the 'No-Free-Lunch Theorem' in ML?

Approach: Explain why no single model is best for all problems.

A theorem stating that if an algorithm performs well on a certain class of problems, it necessarily pays for that with degraded performance on the set of all remaining problems. No single algorithm is universally superior.

Follow-up: What does this imply for the importance of feature engineering?

Key concepts: generalization, prior knowledge, optimization.