tech
The Case Against Skills
Use them intentionally, or they could make your AI worse
TL;DR
- Elaborate custom skill libraries, popularized on platforms like X, may not be improving AI performance and could be making it worse.
- Frontier AI models are often capable of reasoning independently, rendering many skills unnecessary and potentially causing confusion.
- Skills can inflate costs and introduce errors when they conflict with a model's inherent training and capabilities.
- Useful skills are typically those that provide specialized guidance, access to private company data, or custom workflow instructions.
- A benchmark test showed that a majority of software-engineering skills had no impact on performance, and some decreased it.
- The utility of skills has a shelf life, especially those designed to patch a model's weaknesses, as new models absorb these capabilities.
- An audit of AI skills is recommended, focusing on keeping those that provide private context, custom tool access, or specific workflows, and retiring those that don't demonstrably improve results.