Featured Sessions
Let Test Security Out of the Box! Data-Driven Program Evolution
Monday, March 13th | 11:15am-12:05pm
Test security is more than one aspect of a testing program, it is a way to obtain high-level data about how your program functions. Test security data can be obtained through data forensics, web patrol, incident reports, site monitoring, audits, etc. Often, these data are separately consumed and have minimal or limited impact. Synthesizing all test security data gathered by a program and using them to inform program-level decisions can be a smart, data-driven way to propel program evolution.
For example, data forensics is often used to assess the efficacy of existing test security measures, identify hotspots for test security threats, initiate investigations, and invalidate scores, but it can also inform program-level decisions about test design, administration modes and schedules, which data to exclude from psychometric analyses and accountability data, and where and when to focus republication or refurbishment efforts.
In this session, presenters from a state department of education assessment program, a certification program, and a test security company will discuss examples of how test security data has driven changes in program design, including policies and procedures, stakeholder communications, and administration decisions, among other things. It's time to let test security out of the box and start maximizing the impact of program data.
Presenters: Sarah Toton, Caveon; Sarah Pelter, Inteleos; Timothy Butcher, West Virginia Department of Education
Alexa, How Can You Help My Testing Program?
Monday, March 13th | 1:10pm-2:20pm
In the assessment field, Artificial Intelligence (AI) is often misunderstood even though it has been used by testing programs for decades. AI is becoming increasingly commonplace from helping streamline candidate registration to detecting suspicious test-taker behaviors during an exam. AI often works behind the scenes enabling everything from customer service efficiencies in things like chatbots to the generation of sophisticated measurement methods such as adaptive testing.
However, with all the assistance and efficiency that AI provides, there are also several emerging legal and ethical concerns that must be considered. Most concerns lie in the unexplainable black box, meaning the “how” AI works is not explainable or understandable.
The purpose of this presentation is to outline the opportunities, threats, and considerations of using AI in assessment. This includes critical considerations such as transparency, impartiality, authority, and the role of discretion in decision making. In addition, there are several considerations related to the defensibility of AI-led decision making; especially as it relates to the justification and rationale of decisions.
Come engage with a panel of psychometricians, business, and industry leaders as they discuss different ways in which AI can enhance an exam program ranging from improving business operations to strengthening the security of your program as well as what to consider before implementing certain AI tools.
Presenters: Jason Fine, Meazure Learning; Chris Beauchamp, Meazure Learning; Kim Brunnert, Elsevier; David Foster, Caveon; David Yunger, Vaital
Are You Prepared to Meet the Demand for Skill-Level Assessment and Validation?
Monday, March 13th | 9:55am-10:45am
In today’s environment of continuous and unpredictable change, organizational success requires innovation, agility and resilience. To accomplish these goals, leading companies are reskilling and upskilling employees and assigning them to projects and roles based on their validated skills, rather than their job titles. They are deploying and redeploying skills throughout the organization in a just-in-time manner.
Skill-level assessments and data are the foundation of this new approach. Equally important, a skills focus reduces bias in hiring; benefits underserved, underrepresented, and underemployed populations; and supports DEI initiatives. And skill-level assessments contribute to rapid reskilling and upskilling to sustain a qualified, national workforce.
Join us to learn more about how employers and workforce organizations are using skill assessments and data, the factors expected to increase demand for this approach, and the opportunities this trend presents for your organization.
Presenters: Lenora Knapp, Knapp & Associates Int'l; Taylor Sullivan, Codility
Differential Impact of Test Center Versus Online-Proctored Delivery on Gender, Ethnicity and Other Examinee Subgroups
Monday, March 13th | 4:25pm-5:15pm
Research has found mode differences between exams delivered in test centers (TC) versus online proctoring (OP). However, until now, the mode effects on subgroups such as gender and ethnicity have not been examined.
This session will shed light on mode effects by demographics: Does the use of OP do anything to lessen the differences? That is, if there is a performance gap between ethnic groups under TC delivery, does it become smaller, remain unchanged, or get larger via OP delivery?
The studies in this session will address these questions in the context of multi-state regulatory exams and a continued professional certification exam, delivered via TC and OP, to candidates of different ethnicities. Exam score, pass rate, testing time, and effect sizes will be presented, as well as gender, age, and educational levels.
This session will interest psychometricians, test providers, and sponsors, as we grapple with the multi-dimensional issue of fairness in credentialing.
Presenters: Jinghua Liu, Pearson VUE; James Davis, Pearson VUE; Hao Song, The National Board of Certification and Recertification for Nurse Anesthetists
Exams Can Be “Delightful”: Reconsidering Accessibility with a Customer Focus
Tuesday, March 14th | 9:55am-11:05am
This workshop will begin with a discussion of Microsoft’s journey to creating a delightful exam experience that is only possible through a close partnership with ITS and Pearson VUE who worked together to identify and prioritize the challenges that are preventing the experience from being delightful and to develop solutions. We will share lessons learned along the way, including best practices and tips, that you can use as you embark on your own accessibility journey.
With that overview in mind, we will then ask participants to share the accessibility challenges that their programs are facing or those that they believe they will have to resolve. After brainstorming these challenges, we’ll vote on the top 5 challenges to tackle together during the workshop. Depending on the size of the audience, we will assign each challenge to one or more groups and ask them to brainstorm possible solutions that programs can implement to address those challenges or additional research that could be done to better understand how to address the problem from a “delightful experience” perspective. Each group will summarize the results of their discussions with the goal of creating a high-level implementation plan that attendees can leverage in their own programs as they, too, move toward more accessible, delightful exam experiences that benefit not only people with disabilities but all candidates.
Presenters: Liberty Munson, Microsoft; Don Tanedo, Microsoft; Pat Hughes, Internet Testing Systems; Joan Lambert, Pearson
Accommodations in the Age of Hybrid Testing
Wednesday, March 15th | 9:35am-10:25am
As people adapt to new ways of working and meeting, there is a strong push to provide workers the flexibility to work and live anywhere. This change will continue to push the shift to greater hybrid assessment opportunities where the test taker can choose whether they test in-person or remotely. The shift to remote testing changes the nature of special accommodations that are requested by test takers and how they might be implemented. Testing programs and exam providers actively seek ways to manage candidate expectations and maintain the fairness and validity of their assessments.
Presenters will share how two organizations partnered to meet the additional challenges of maintaining content security and identification confirmation while providing accommodations support in remote exam administrations. They will take the audience through the challenges and issues and show how technology provided remote support for the most complex accommodations, including readers, scribes, translators, and more while maintaining content security and verifying test-taker identity. The audience will be introduced to innovative features including content blur, screen, and keyboard control, and much more. Leave the session with a solid understanding of what a program should consider when adopting technologies in a hybrid testing environment.
Presenters: Stephanie Dille, Meazure Learning; Ashley Norris, Meazure Learning; Andy Dua, Uexams; Carl Bowman, CompTIA
Legal Eagle Update (and Q&A)
Monday, March 13th | 9:55am-10:45am
This session will include a Legal Eagle Panel with a combined total of 95 years of legal experience that will provide an update on the most pressing legal issues facing testing organizations, following by a Q&A session. The Legal Eagle panel will include four attorneys, each from a different law firm, with extensive experience representing testing organizations and service providers. In the first 25 minutes of this session, each member of the panel will provide a “legal update” on important legal issues in testing, including ADA accommodations, copyright registration and enforcement for secure tests, trade secret law as it pertains to test content, contract terms and enforcement, and invalidation of test scores. In the last 25 minutes, the panel will answer questions from the attendees. If there was ever a question you wanted to ask a lawyer about testing, this is your opportunity!
Presenters: Robert Burgoyne; Jennifer Ancona, Semko, Baker & Mckenzie; Allison Mulford, Caveon, LLC; Marc Weinstein, Marc J. Weinstein PLLC
Strategic Insight: Adapting GDPR to China’s PIPL: From Test Administration to Certification
Monday, March 13th | 11:15am-12:05pm
Following the EU GDPR taking effect in 2018 on personal data privacy regulation, countries around the globe, such as China have implemented their own data privacy mandates. ATP recently published a Privacy in Practice Bulletin, “Complying with PRC Personal Information Protection Law” (‘PIPL') to help industry stakeholders better understand China’s PIPL.
In this session, Panelists from Asia ATP and the International Privacy Subcommittee discuss the impacts, preparation and strategies of implementing PIPL compliant data privacy processes by drawing on GDPR lessons. First, we provide an overview of China’s PIPL. Then an expert panelist will provide a comparison between the EU GDPR and China’s PIPL as a basis for discussion. The panel discussion will be focused on vendors’ responses from certification/licensure, education, workplace assessment, and personality testing perspectives. We will discuss how companies are working to collect and show evidence of PIPL compliance, including online test administration, remote proctoring, data collection and storage, and cross-country data transfer certification, in different domains of testing. Finally, panelists will discuss how to comply with the Chinese data privacy laws and regulations to turn business challenges into attainable opportunities.
Presenters: Changhua Rich, ACT; Alex Tong, ATA, China; Gary Behrens, FifthTheory, LLC; Krista Pederson, Hogan Assessment Systems, Asia Pacific
Threat or Opportunity? The changing landscape of assessment.
Monday, March 13th | 4:25pm-5:15pm
The assessment industry is facing new threats to the use of assessment as part of entering professions, qualifying for medical reimbursement, etc. but, we are also becoming armed with new ways to broaden our reach and better serve new audiences: microcredentials, assessment-based certificates, technologies enabling more interesting test design & delivery. Are we facing irrelevance or increased ability to serve? The panel will provide an overview of the various challenges to our traditional model and roles, along with recent advances and opportunities to expand our reach.
Takeaways:
- Examples of current threats
- Understanding of emerging technologies for test assembly, analysis, and administration
- Understanding of new exam types, associated standards, and current uses Ideas for broadening scope & reach for testing organizations
Presenters: Donald Balasa, American Association of Medical Assistants; Kellie Early, LSAC; Frank Williams, PSI Services, Credentialing; Criag Mills, Mills Strategy Partners; Christine Mills, Ascend Learning; Bill West, Cornerstone Strategies
What’s new with the new? A conversation on key opportunities and what is in store for the testing industry.
Tuesday, March 14th | 11:35am-12:25pm
The past year has been prolific in leadership changes and new vision in our testing industry. How about we take a moment to chat with a few of the new CEOs and discuss their perspective and vision for the future of our industry? From innovations to growth, business strategy and key opportunities, let’s get in the head of these leaders and learn from their experience as well as understand where they see our industry going to ensure it continues to thrive and evolve. Even better, come prepared with the questions you always wanted to ask to these great leaders, and get the most out of having them assembled together for this panel conversation.
Presenters: Tim McClinton, Meazure Learning; Gary Gates, Pearson VUE; Joy Jones, GMAC; Isabelle Gonthier, PSI Services; Pierre Le Mahn, PMI
Real World Implementations of Two, New Collusion Statistics: Approximation Response and Score Similarity Analyses
Tuesday, March 14th | 11:35am-12:25pm
Exam security continues to be a primary concern for credentialing organizations. Many organizations struggle to find a solution that is near real-time, automated, and actionable. This panel presentation will describe a relatively easy and efficient method to identify potential collusion. The method is meant to approximate score or response similarity (SSI or RSI) analyses with relatively simple calculations. SSI and RSI are typically computationally intensive, require specialized software and are based on relatively strong assumptions.
During this session, the panelists will begin by providing an accessible introduction to the two new, simplified similarity indices, Approximation RSI and SSI. The remainder of the session will focus on the application of each approximation to two distinctly different certification programs. Specifically, each certification organization will discuss the pros and cons of implementing Approximation RSI and SSI within their programs, how and when these analyses are actionable, and if and how these analyses have changed their approach to security. This session will leave participants with increased knowledge and tools to support timely, computationally less intensive, and actionable results that can be implemented at their own organizations.
Presenters: Corina Owens, Alpine Testing Solutions; Russell Smith, Alpine Testing Solutions; Patrick Irwin, Palo Alto Networks; Danielle Castelli, CompTIA; Carl Bowman, CompTIA
Tackling Test Fraud – From Detection to Action
Wednesday, March 15th | 8:30am-9:20am
In a relatively short period of time, the pandemic accelerated the adoption of online remote high stakes testing to support virtual learning, credentialing, and talent acquisition practices that became necessary for public safety and economic continuity. Many high stakes testing programs have now become multi-modal with online remote proctored and test center delivery. As a result, the focus on test security has intensified and organizations are reviewing their processes to ensure that security risks are addressed, including various forms of test fraud (e.g., cheating, content theft). In this session, a diverse panel of experts in high stakes assessment program management, security operations, and psychometrics will discuss key considerations for test security in the context of multi-modal high stakes testing. The panel will explore security risks and operational considerations for taking action when test fraud is detected with the aim of ensuring the integrity of assessment.
Presenters: John Weiner, PSI; Kim Cohen, ISACA; Carl Bowman, CompTIA; Nichole Tucker, PSI; Jeff Marsh, Ascend Learning
The Psychometric Challenges of VR
Wednesday, March 15th | 9:35am-10:25am
This session describes how to design the job task analysis(JTA) in a way that captures the physical and cognitive behaviors required so VR can accurately score the execution of those behaviors. The session also explains how to use criticality data to program the weighting of scores when errors threaten safety, injury, loss of life, or other risks.
Presenters: Judith Hale, Center for International Credentials, LLC; Mike Christian, From the Future; Rose Baker, University of North Texas
An “Empathy Deficit”?: How Can the Testing Industry Help People to “Get Along”
Monday, March 13th | 9:55am-10:45am
Bullying rates are up (particularly for individuals seen as different and/or from historically underrepresented groups), workers are feeling isolated, and a recent SHRM report lamented that the pandemic has resulted in a society with an “empathy deficit.” These concerns have implications for individuals’ sense of belonging, their ability to cope effectively with their environment, and ultimately their health and well-being.
This session asks: How can the testing industry help people and organizations to address this empathy deficit (and to develop other essential skills)? Panelists will lay out the argument that assessment organizations already have a variety of tools to help people measure and understand their personality traits, social and emotional skills, and mental health and, through an asset-based approach, can use the insights from these tools to help people develop more adaptive behaviors, better ways of working together, and a sense of well-being.
Presenters: Alex Casillas, ACT; Joanna Solomon, MHS; Chase Winterberg, Hogan Assessments; Shaun Wehle, Talogy
View the Full Program