Welcome to episode six of Tails of Testing! Sandy Hogg, our Business Manager, chats with Ryan Howard, our Solutions Portfolio Manager and expert on continuous learning. Continuous learning, also known as longitudinal assessment or continuous recertification, is shaking up traditional models of recertification, shifting them from infrequent, high-stakes assessments to frequent, lower-stakes assessments. Learn about how this new model embeds learning with assessment, its many applications in the industry, and more in this video!

Video Transcript

[Onscreen: A split-screen of Ryan (left) and Sandy (right) chatting virtually.]

Sandy Hogg
Hi, everyone. Thank you for joining us for another episode of Tails of Testing. My name is Sandy Hogg. I’m the Business Manager here at ITS, and I’m joined today by my colleague Ryan Howard, Solutions Portfolio Manager at ITS. Welcome to Tails of Testing, Ryan. Can you introduce yourself?

Ryan Howard
Super excited to be here. Thanks, Sandy. And thanks for the opportunity to talk about continuous learning. My name is Ryan Howard. As Sandy said, I’m the Portfolio Manager at ITS, and my main responsibility here is really working with our partners to implement and grow their longitudinal assessment and continuous learning assessments.

Sandy
Awesome. Thanks, Ryan. All right, so, we’ve got some exciting conferences coming up this fall: the ABMS Conference and ICE. And I know that continuous learning is going to be a popular topic at both of these conferences. I feel like actually there have been sessions about continuous learning over the last several years at ATP and other industry events. So, before we go any further in this discussion, can you tell us, Ryan, what exactly is this concept of continuous learning that we keep hearing about, and why are you excited about it?

Ryan
Well, great question. So, continuous learning, or some of you may have heard it referred to as continuous recertification or continuous practice, is our, ITS’s, next-generation system for practice recertification and test prep. So, traditionally, many recertification or maintenance of certification programs were designed to be delivered at testing centers years apart, let’s just say every 10 years. Well, technology, medicine, just the world in general is advancing so quickly that it really doesn’t make sense to gauge someone’s skills or abilities once every 10 years. So, what this system does is it helps testing programs, you know, move away from that point-in-time assessment and instead spaces out that assessment over time and provides more flexibility to take questions on the go. The system also—I’m super excited about this part—really deeply embeds learning opportunities, learning content, and candidate visibility into their performance within the assessment to make the assessment more engaging. You know, test takers are not just assessing their knowledge and demonstrating their proficiency, but they’re also identifying and filling knowledge gaps, which really personalizes the experience.

Sandy
Wow, that’s sounds wonderful. You know, I hear you say things like flexibility and engagement and, you know, also spacing out assessment. It sounds like it takes some of the pressure off, too. So, I’m really interested—you mentioned learning here. I’d love to hear more about this learning aspect. You know, you said the system embeds learning with assessment. What does that mean exactly?

Ryan
Well, it can mean a lot of things to a lot of different people, so you know this concept is really at the core of continuous learning. You know, our system has many, many configuration options for partners to choose from of how they want to promote learning or integrate learning through assessment. So, take our adaptive algorithm as an example. If test takers will get a question incorrect or answer a question with, say, low confidence, they might be presented with an opportunity to retake that question, or retake a variation of that question, or retake a question in that same content area later to try to reinforce that learning experience. You know, they get to spend more time on topic areas that they might be struggling with and see how their performance improves over time. We have an out-of-the-box learning algorithm and we also support tools for programs to build their own learning algorithms for redelivery of items. But then another way to promote learning is to integrate supplemental learning content directly into the testing platform. So, this content can be in the form of journal articles, videos, study guides, key learning points, summaries. You know, the sky is really the limit. And this content can be made available directly in the test at test questions (answer a question and you get to learn about that question), as well as outside of the test and can tie back to that specific question or that topic area that the candidate is about to see or they can see that they’re struggling in. So, we’re really excited to support many different continuous learning programs because it’s really showing that learning and assessment can run in parallel.

Sandy
Wow, yes, more like two sides of the same coin rather than, you know, different things. So interesting. So, this type of testing model does seem like it could have a lot of applications, you know, I know you’ve mentioned, you know, practice and recertification. So, I’m really curious to know, how is it being used today?

Ryan
Well, we’ve seen a lot of medical boards really embrace this model. You know, there’s many other opportunities for, you know, this model, but medical boards have really jumped on it, where instead of their traditional and expensive once-in-10-years maintenance of certification assessment that nearly every medical board requires, the boards are opting for a more lower-stakes, longitudinal approach. So, for example, every board takes it a little bit differently, which is great because one size doesn’t fit all, but basically, you know, their candidates answer a certain number of questions periodically. Let’s just say quarterly from the comfort of their own home or office. We approach this platform with mobile first in mind. So, boards might want a specific set of questions to be delivered in each period. Some boards might allow for candidates to select which topic areas will be included in their assessment, whereas other boards might opt for a more adaptive approach where items are pulled from a larger pool, and candidates, you know, specifically based off of candidates’ performance and their specific feedback on the item. So, the system is really configurable and supports a wide range of program design, so the possibilities are really endless. But one of the benefits of this shift—at least we’re seeing in the medical industry—is that allows boards and candidates, test takers, to keep the content, to keep the learning material, more up to date with constantly evolving concepts. You test once every 10 years, you’re just proving that you know that year. If you’re testing nine questions a quarter, and those quarters are written every quarter or every six months or every year, you know, you’re able to, you know, stay relevant in your industry.

Sandy
Wow, and that seems pretty important for the medical industry, right? Wow. Great. That’s really interesting that this, you know, continuous learning testing model can take on such different forms even within the same industry, all the things you were listing off. I’m sure that we could talk for hours about the different kinds of program designs that you’re seeing, but, you know, that’s a conversation for another day, Ryan. All right, so we’ve learned a little bit more about continuous learning and how it’s being applied today. So, I’d love to know, you know, what’s to come? What’s next for this assessment model?

Ryan
Yeah, I really think continuous learning is the way of the future, not just for the medical space. You know, we’re always talking to our partners and keeping a lookout for new features to integrate into our product suite. You know, some things on the horizon are integrations with other learning management systems or generating study materials such as notecards based off of assessment questions. You know, and since the concept is really building on the foundation of learning and assessment, the sky is the limit.

Sandy
Oh wow, that all sounds really exciting. Very cool. We’ll keep an eye out for all of those things. All right, so we’re just about out of time, but before we wrap up, do you have anything that you want to leave this audience with?

Ryan
That’s a great question. I think the one takeaway is acknowledging that any transition to a new testing model or methodology can be really intimidating, but the benefit of this model is that you can determine how simple or complex you want your continuous learning program to be. You know, you don’t have to have hundreds of items in your item bank in order to use a continuous learning platform. You can start out small with a fixed form of questions and perhaps scale up from there. Maybe one day you’ll have a, you know, a robust item pool with hundreds of items in it and move to dynamic item delivery. But you don’t have to start there, and you don’t have to end there. You’re not locked into one design, and we’ll work with you to determine what is the right fit for you. And if you’re curious to know more, again, I could talk about this for days. Come find me at the upcoming ABMS and ICEE conferences this fall. Either come to the booth, come to one of our sessions, stop me in the hallway. I’m always happy to talk about this.

Sandy
Wonderful. Well, Ryan, that’s great advice. You know, thank you once again so much for joining us today. Really appreciate your time and looking forward to the next time we get to chat.

Ryan
Thank you, Sandy.

Sandy
Thanks.

About Our Guests

Sandy is the Business Manager at ITS and has over a decade of experience in program management and business development. Her strong background paired with her natural curiosity and solutions-oriented mindset allows her to develop and implement efficient processes and tools that support the Sales and Marketing team. Sandy is also an essential part of the proposal team and product demo process. Outside of work, you can find her running long distances or hiking with her family.

Ryan is the Solutions Portfolio Manager at ITS and serves as the go-to specialist for the Continuous Learning and Longitudinal Assessment programs. His extensive experience in the industry enables him to understand clients’ needs and communicate their priorities across ITS’s teams so that he can strategically implement, manage, and grow their programs. Outside of ITS, Ryan enjoys completing DIY home projects and going on outdoor adventures with his dog, Payson.