The Critical Intersection of Assessment, Psychometrics, and AI in Workplace Learning
In a recent episode of the "Future of Learning" podcast from Open eLMS, host Andy Howie welcomed Nathan Thompson, a PhD psychometrician and CEO of Assessment Systems, for a compelling discussion on the often-overlooked world of assessment. Thompson, whose company is behind tools like FastTest, brought nearly two decades of expertise to the conversation, highlighting the profound impact of assessment quality on organisational decision-making and the emerging influence of artificial intelligence.
The discussion underscored a critical point: "bad tests make bad decisions at scale every day." This insight resonates deeply within the Learning and Development (L&D) sector, where the effectiveness of training and development programmes hinges on accurate measurement. Thompson’s perspective, rooted in psychometrics – the science of assessment – offered a unique lens on how organisations can ensure their evaluations are not just routine, but truly robust and fair.
Key Takeaways from the Discussion:
- The Damage of Poor Assessment: Bad tests lack reliability (precision) and validity (measuring what they intend to) and can lead to flawed hiring, certification, and educational outcomes.
- Why It Goes Unnoticed: A lack of "assessment literacy" within organisations, budget constraints, and ingrained habits ("it's always been done this way") often prevent the adoption of more scientific assessment methods.
- Hiring Bears the Brunt: Thompson identified hiring as the sector carrying the most invisible cost from poor assessment, particularly in smaller organisations relying on subjective methods.
- AI's Impact on Assessment: Artificial intelligence is significantly transforming item (question) development, test assembly, feedback, and essay grading, making these processes more efficient.
- Scepticism is Key: While AI offers powerful tools, psychometricians must remain sceptical, prioritising validity and ensuring human oversight ("human in the loop") in high-stakes assessments to guard against AI hallucinations or misuse.
- Shifting Professional Roles: AI is taking over mundane, entry-level tasks in assessment, allowing psychometricians to focus on higher-level design, validation, security, and strategic management of assessment programmes.
- Bridging Silos: Workforce pressures are forcing L&D, assessment, and credentialing to converge, requiring greater collaboration despite necessary firewalls for certification integrity.
- Building Trust in Credentials: For credentials to be truly trusted by employers, they often need to be legally required or accredited by recognised international bodies.
- Assessment as a Science: L&D leaders should approach assessment scientifically, understanding principles of reliability, validity, and cut-score setting, leveraging new tools and increased assessment literacy.
The Silent Damage of Bad Assessments
Nathan Thompson opened the discussion by stating that bad tests lead to bad decisions. He defined a "good" test by two core psychometric principles: reliability and validity. Reliability refers to a test's precision and its ability to produce consistent, repeatable scores. Validity ensures that the scores genuinely support the intended interpretations – for example, that a person has mastered content or is proficient enough for a profession. Without these, tests can be unreliable, producing random numbers, or invalid, testing the wrong things, having unfair cut scores, or being biased. Psychometricians, Thompson noted, are dedicated to making tests fairer and more equitable, not just harder.
Host Andy Howie probed why these issues often go unnoticed. Thompson cited several reasons, including a lack of "assessment literacy" where organisations simply aren't aware of the science behind good assessment. Advanced techniques like item response theory (IRT) and computerised adaptive testing (CAT), despite being decades old, remain underutilised due to budget and bandwidth constraints in smaller organisations. Even foundational classical test theory, involving simpler mathematics, is often overlooked. Other factors include legal requirements dictating assessment methods without adequate budget for improvement, and the pervasive "it's always been done this way" mentality.
When asked which sector carries the most invisible cost from bad assessment, Thompson pointed to hiring. While large corporations like Amazon might have sophisticated in-house assessment departments, countless smaller businesses rely on less formal, often subjective, methods like interviews and resume reviews, leading to significant, unquantified costs from poor hiring decisions.
The Science Behind Sound Assessment: Psychometrics Unveiled
To clarify the field, Thompson defined psychometrics as the science of assessment, applicable across employment, education, certification, licensure, and even medical questionnaires. He explained that it involves both data science – analysing item difficulty, identifying confusing questions, and equating scores across different test administrations (a common challenge in UK exams like GCSEs and A levels) – and procedural aspects, such as convening expert panels to set fair cut scores. This rigorous approach counters public perception that exams are simply getting easier, by ensuring comparability and fairness over time.
AI's Transformative Role in Assessment
The conversation naturally shifted to artificial intelligence. Thompson highlighted that AI has made the most significant impact on item (question) development, traditionally one of the most labour-intensive parts of the assessment process. Studies have even shown AI-generated items can have higher acceptance rates than human-generated ones, depending on complexity and available documentation. Beyond item generation, AI is proving valuable in assembling new test forms, producing reports, giving feedback, and grading essays.
However, Thompson stressed the importance of scepticism. For psychometricians, validity remains paramount. Any AI application must be supported by documentation and research demonstrating its effectiveness. He cautioned against the "magic silver bullet" mentality, citing an example of an organisation wanting AI to grade essays without providing a rubric or even defining grading criteria. AI, he argued, requires clear guidance, examples, and human oversight, especially for high-stakes assessments.
When discussing specific AI tools, Thompson noted Claude's strength in textual tasks like item writing and review, even observing it generating its own code for item analysis. Gemini, while less consistent with text, performs better with image generation. He encouraged L&D professionals to experiment with these tools and consult research to find the best fit for their needs.
Navigating the AI Landscape: Scepticism and Strategy
Thompson believes AI represents a genuine structural shift, not just cyclical hype, evidenced by its widespread adoption and discussion at industry conferences. Unlike previous technologies like blockchain, which failed to gain traction in assessment, AI is actively being integrated.
The impact on the psychometrics profession is significant. Item writers are affected first, as AI automates much of the mundane work. However, Thompson draws parallels with other professions, where AI takes over entry-level tasks, allowing experienced professionals to focus on higher-level, strategic work. For psychometricians, this means shifting towards designing tests, validating them, ensuring security, and managing the overall assessment process – areas where human judgment remains critical. While it may initially affect the number of people in the profession, it could ultimately expand the market for assessment.
Organisations risk getting the transition wrong by viewing AI as a complete solution or assuming they need highly specialised, custom-built AI software. Thompson advised starting with readily available tools like Claude to experiment and understand their capabilities before investing heavily.
For those entering the field, building judgment is crucial. While AI handles routine calculations and item generation, new psychometricians must still understand foundational concepts and gain practical experience in managing expert panels, setting cut scores, and designing assessments from the ground up – skills often not fully covered in academic programmes. Thompson warned of the risk of losing the "feel for good measurement" if professionals rely solely on automated tools without understanding the underlying principles.
Bridging the Silos: L&D, Assessment, and Credentialing
Thompson's experience turning around struggling companies highlighted the stress organisations face when they fail to modernise, clinging to outdated processes. This extends beyond assessment development to the entire business of assessment, including application processing, recertification, and managing continuing education.
The discussion also touched on digital credentials (e.g., Open Badges, Credly). Thompson welcomed the broader push for education and credentialing but clarified that these platforms are tools for credential sponsors (like Amazon or Salesforce), who remain responsible for defining and running the underlying tests. For credentials to be truly trusted, he emphasised the importance of accreditation by international bodies such as NCCA and ISO ANAB, which provide a stamp of approval for best practices.
Historically, assessment, learning and development, and credentialing have often operated in parallel. Thompson argued that workforce pressures, such as demographic shifts, are now forcing these worlds to converge. While legal firewalls might separate education and certification departments to prevent conflicts of interest, internal collaboration is essential for recruitment, learner preparation, and ensuring individuals stay in their professions.
For a credential to move beyond a "nice to have" on a CV to something an employer genuinely trusts, it must either be legally required or accredited by a reputable body. Competition in the job market also plays a role: in professions with a surplus of candidates, a robust certification can provide a significant advantage.
Building Robust Assessments in the AI Era
When asked what he would change first in an L&D leader's approach to assessment, Thompson urged them to view it as a science. He highlighted the extensive effort that goes into major assessments like A levels, contrasting it with simple, quickly drafted quizzes. With the increased accessibility of assessment psychometrics via online resources and AI, L&D professionals can now easily learn about reliability, validity, and cut-score setting.
For a genuinely worthwhile and robust assessment, Thompson advised following the guidelines of accreditation bodies or other best practice organisations like the International Test Commission. These typically involve job analysis, test design, item writing aligned to blueprints, piloting, and formal cut-score setting processes. While not every organisation needs to implement every guideline, selecting key elements can significantly improve assessment quality.
Thompson concluded with a powerful takeaway: the importance of documentation. Having clear documentation for assessment decisions – test blueprints, cut-score rationale, and annual statistical reports – provides crucial evidence of validity. This not only ensures the integrity of the assessment but also offers a defensible position should an assessment ever be challenged. Even a few hours spent defining these elements can provide significant protection and demonstrate a commitment to fair and effective assessment.
Open eLMS champions the integration of robust assessment practices within modern learning ecosystems. Our cloud-based Learning Management System (LMS) provides comprehensive tools for live learning, learning tracking, skills monitoring, and custom reporting, ensuring that every aspect of your training programme is measurable and effective. Furthermore, the Open eLMS Learning Generator, powered by AI, enables organisations to create custom eLearning courses, videos, and gamification in minutes, facilitating the rapid development of high-quality, assessable content.
Discover how Open eLMS can transform your organisation's approach to learning and assessment, ensuring your investment in L&D delivers tangible, measurable results.
Explore the Open eLMS LMS and the Open eLMS Learning Generator today at https://www.openelms.com.