I paid people to try and follow my README

User-tested documentation is now a demonstrable improvement over AI-generated guidance for software installations.

Terence Eden paid individuals €25 per hour to test the installation process for his ActivityBot project, specifically focusing on the clarity and usability of its README file. During these sessions, testers shared their screens and vocalized their experiences, highlighting confusion, errors, and frustrations. Eden documented these issues, which included a broken demo link, unclear terminology, and assumptions about user knowledge regarding file operations and server permissions. He iterated on the README based on this feedback, retesting with subsequent participants. In total, Eden spent approximately €150 to gather this direct user feedback. He advocates for this hands-on testing with real people over relying solely on AI simulations for improving documentation.

The experiment underscores a critical gap in developer-centric documentation: the inherent bias of creators. Eden’s direct payment for user feedback, totaling €150, provides a tangible alternative to internal testing or LLM simulations, emphasizing the irreplaceable value of human interaction and real-world problem-solving in refining user experiences. This approach could signal a shift towards more rigorous, human-centered validation for software documentation, especially in open-source and community-driven projects. Companies and independent developers alike may need to re-evaluate their documentation strategies, investing in direct user testing to ensure clarity and accessibility for their target audience. The insights gained, such as the confusion around simple commands like file renaming and the non-uniformity of web server permissions, highlight that even seemingly basic elements require user validation.

The success of Eden's method suggests that actively seeking and incorporating critical feedback from real users, even if it means paying for their time, is a more effective path to polished documentation than relying solely on automated tools or internal reviews. This focus on 'seeing their cat' and hearing their frustration points to a broader need for empathy and direct communication in the software development lifecycle. As AI writing assistants become more prevalent, the ability to communicate complex technical information effectively to human users will likely become an even more valued skill and a key differentiator for successful software projects. Developers and project maintainers should consider allocating resources for such user testing, potentially seeing it as an investment that significantly reduces support overhead and improves user adoption.