Did the Training Actually Work? What New Research Says About Measurement, AI, and Transfer

by Chris Packert | Oct 17, 2026 | Featured, Learning Strategy

Emerge ID: Award-winning learning design. AI-powered development. Smarter training at a lower cost.

Dashboards, badges, interactive activities, AI tools. Each one sounds like progress. In Episode 2 of our podcast, L&D Insights at Altitude, our hosts Nora and Ben walk through seven current sources (four peer-reviewed studies, a Gallup study, an OECD report, and a new Colorado tax law) and ask the same thing of each: what changes in how people work? This week's thread is simple. Features are not learning. Each one works only as well as the design around it. Here is what our team took from them.

Watch or listen to Episode 2 on YouTube

Enjoying the show? If you like our podcasts, be sure to like and subscribe to our YouTube channel.

Dashboards: a small average and a wide range

In The Internet and Higher Education, Rivan Gestiardi and Yuniawatika ran a meta-analysis of learning analytics dashboards, displays of learners' own data such as progress or activity. They pooled 21 of 22 controlled studies with more than 33,000 participants, and they left out dashboards bundled with adaptive tutoring or heavy automated feedback, so the result reflects the dashboard itself. For learner-facing dashboards, the average effect was small, about 0.29, and closer to 0.17 after adjusting for possible publication bias. The range across studies was wide enough to include zero. Effects showed up for grades and completion, but not for engagement logs, self-reported self-regulation, or motivation.

The authors' own point is that a given rollout may yield no benefit, so pilot with usage data and evaluate before you scale. The participants were mostly higher education students, not employees, and the authors rate certainty as low to moderate. The screen is not the intervention. What people do with it is.

Badges that backfired

In Behaviour & Information Technology, Sebiha Balci and Bradley Morris randomly assigned 162 online learners to three versions of a course: badges they knew were coming, surprise badges, or no badges. The hypothesis was that surprise badges would work better. It did not hold. Both kinds of badges reduced intrinsic motivation, did not meet learners' need to feel competent, and did not improve performance compared with no badges. It is one short-term study with online students, but it is randomized, and it is a good reason to test badges before you switch them on everywhere.

"Interactive" is a label, not a result

"Interactive beats passive" is a common pitch line. In Learning and Instruction, Christina Wekerle, Michael Sailer, Ingo Kollar, and Tim Fütterer tested the ICAP framework with 249 future teachers. After reading a text, participants reread it, underlined it, answered open questions alone, or answered the same questions in pairs. No condition won on knowledge: the authors found no significant differences in surface or deep knowledge. The pairs did report a stronger sense of belonging. This was one reading task with trainee teachers, and the authors do not say the framework is wrong. They say its evidence base may not be robust yet. Calling something interactive does not make it effective. The task has to make people think and practice.

How good designers work with AI

In the International Journal of Educational Technology in Higher Education, Fang, Feng, Yan, and Zhou watched 67 pre-service teachers build lesson plans with an AI assistant, then rated the plans. The stronger plans came from people who kept cycling between evaluating the AI's draft, revising it, and evaluating again. The weaker plans came from people stuck in loops of unclear back-and-forth. The stronger group also reported lower mental demand and frustration. It is one program of trainee teachers, so treat it as an early signal for workplace designers, and it is correlational: the groups were split by outcome, so it does not prove the habit caused the quality. Still, evaluate, revise, and re-evaluate is a habit you can coach.

Who gets value from AI at work, and who is left out

Gallup's first results from year two of the American Job Quality Study draw on a survey of more than 15,000 U.S. employees. Among employees who use AI, 63 percent say it helps them work faster and 56 percent say it helps them find more creative solutions. Regular use is concentrated, though. College graduates are more than twice as likely to use it daily or weekly (40 percent versus 17 percent), and so are managers compared with individual contributors (37 percent versus 25 percent). In healthcare support, manufacturing production, and retail sales, 70 to more than 80 percent never use AI in their role. Gallup is clear that these are employees' own assessments and that the link to job quality needs more research.

Our reading, not Gallup's claim: if AI training goes mainly to the people already using AI, it can widen the gap.

Most adults are not asking for more training

The OECD's Education at a Glance 2026, released September 29, includes a chapter on adult learning that draws mostly on the 2023 Survey of Adult Skills. Over 75 percent of adults report no interest in more non-formal learning beyond what they have already done. Participation is unequal, too: about 49 percent of employed adults took part in the past year, versus about 34 percent of unemployed adults. The OECD suggests testing interventions that target motivation, not only access. The data are self-reported and cross-country, and the U.S. is one country in the mix. For L&D, the message is plain. Another catalog will not fill seats. Learning has to connect to work people already care about.

Colorado's new software tax and your courses

In June, Governor Jared Polis signed HB26-1223. Starting January 1, 2027, Colorado's state sales and use tax covers computer software however it is delivered, including downloads, software you reach over the internet, and mobile apps. The Colorado Department of Revenue's Computer Software guidance, revised in October 2026, walks through what it means.

The state rate is 2.9 percent. Local taxes are less clear. The legislature's final fiscal note says counties, most cities, and districts like RTD only add this tax if their own voters approve it, and about 70 home-rule cities run their own sales taxes and make their own call. Sellers based in Colorado collect the tax. Out-of-state sellers start collecting once their Colorado sales top $100,000 a year, which was already the rule. If a seller does not collect, the Colorado buyer generally owes use tax instead.

Why does this matter for training? The law keeps two exemptions: software developed for a particular user, and software under an individually negotiated license that both sides sign in writing before or when access begins. Click-through terms and standard form contracts do not count, even if someone signs them. That means a custom course built for one client and a catalog course bought at a website checkout could be treated differently. The law does not mention courses, and the state has not said when an online course counts as software.

Before January, take two questions to your tax advisor. Was this course built for us alone? And is our license individually negotiated and signed by both sides? How you contract and deliver a course may change what you owe.

This episode is general information, not tax, legal, or accounting advice. Emerge ID is not a CPA firm or law firm. Colorado's rules for HB26-1223 are still being written. Consult a qualified tax advisor about your situation.

One contrarian take

The popular story is that better learning tech means better learning: add the dashboard, add the badges, make it interactive, give everyone AI. This week's sources say otherwise. Dashboards helped a little on average and maybe not at all for a given rollout. Badges cost motivation in one randomized test. Interactive activities did not move knowledge in another. Strong AI-assisted work came from judgment, not prompting. And most adults are not asking for more courses.

The feature is the easy part. The design is the hard part: what people need to do differently, why they would care, the practice and support on the job, and how you will measure it. At Emerge ID, Intelligent Design is what the ID in our name stands for. It is what we call our process for combining proven instructional design principles with the speed and production power of artificial intelligence.

Five takeaways

  1. Pilot dashboards and measure before you scale. The average effect was small, and the range included zero.
  2. Test badges before you turn them on. They backfired in one controlled study.
  3. Treat "interactive" as a label. Design the thinking and practice the task requires.
  4. With AI, coach the evaluate and revise loop. Have learners judge an AI draft, revise it, then judge it again.
  5. Aim AI and learning at the people who are not getting them yet. Start with why the learning matters to their work.

The practical challenge

Pick one feature you added to a program this year, such as a dashboard, a badge, or an AI tool, and ask what changed in how people work because of it. If you do not know, that is the place to start.

The ID Angle: what this changes on Monday

Start with the behavior, not the feature. Write down what people should do differently on the job, then pick the tool that supports it. Treat each feature as a test, not a default: pilot a dashboard with a comparison group, and try badges small before you roll them out. Build AI practice around judgment. Open with why the learning matters to people's work. And for Colorado, know how your courses are contracted and delivered, and take the two exemption questions to your tax advisor before January.

Let's design learning that changes what people do

As the episode puts it, "Anyone can add a feature in an afternoon. The hard part is designing learning that changes what people do on the job." That is what Intelligent Design is for: starting with the performance you need, then using proven instructional design principles and the speed of AI to build learning that holds up on the job.

Emerge ID is led by a United States Olympic and Paralympic Committee Coach Educator of the Year with 30+ years of experience in learning and development. If that sounds like your team, there is an easy next step: book a free 20-minute review with Chris Packert. By the next business day, you will get three prioritized next steps. No purchase required.

Planning a new course, onboarding program, or training pathway? Consider Emerge ID for your next project.

Find more episodes of L&D Insights at Altitude every two weeks.

Show notes: Episode 2 chapters

Runtime 13:01. Chapter times match the YouTube video.

  • 00:00 Cold open: do dashboards help people learn?
  • 00:24 Welcome to L&D Insights at Altitude
  • 00:53 This week's thread: features and learning
  • 01:17 Dashboards: a small average and a wide range
  • 02:33 Badges that backfired
  • 03:23 Interactive beats passive? Testing ICAP
  • 04:25 How good designers work with AI
  • 05:28 Who gets value from AI at work? (Gallup)
  • 06:25 Most adults aren't asking for more training (OECD)
  • 07:18 Colorado's new software tax and your courses
  • 09:31 One contrarian take
  • 10:25 Recap
  • 10:51 This week's challenge
  • 11:05 The ID Angle
  • 12:15 Next steps

Hosted by AI voices. Research by the Emerge ID team.

Gallup's American Job Quality Study is conducted with Jobs for the Future and The Families & Workers Fund.

Sources cited in Episode 2

Colorado HB26-1223 sources:

Ep. 01 header for L&D Insights at Altitude: "Did the Training Actually Work?" over a night sky and Rocky Mountain ridgeline.

Emerge ID: Award-winning learning design. AI-powered development. Smarter training at a lower cost.

In this episode, our hosts Nora and Ben walk through seven current sources, one professional body report and six peer-reviewed studies, and ask the same thing of each: does it actually change how people work? Here is what our team took from them.

Watch or listen to Episode 1 on YouTube

Enjoying the show? If you like our podcasts, be sure to like and subscribe to our YouTube channel.

Employers are building talent, so "people attended" is no longer enough

The Chartered Institute of Personnel and Development released its Resourcing and Talent Planning Report 2026 (Hogarth and McCartney) in September, surveying 1,014 UK HR professionals in resourcing and talent planning. Forty-eight percent of employers increased their efforts to develop talent in-house over the past year, and 91 percent are already upskilling existing employees or would consider it to fill hard-to-recruit roles. Among organizations that acted on retention, increasing learning and development opportunities was the most common step, at 45 percent.

Building rather than buying raises the bar: when learning fills skill gaps, completing a course only matters if people can now do the job.

Proving it depends on the measure, and a review in Human Resource Development Quarterly shows how easily measures go wrong. Josef Guggemos and Michael Goller reexamined 325 instruments from 92 workplace learning studies. In a reflective measure, each item is a sign of one underlying quality, so the items should rise and fall together. In a formative measure, each item is a distinct ingredient, and together they make up what is measured, so dropping one changes the score's meaning. Depending on how strictly the authors coded them, about 35 to 50 percent of instruments were formative by design but analyzed as reflective. Analyzing one kind as the other can bias the results. The lesson for practice: before you put a survey score in front of your CFO, ask what the instrument measures and how it was built.

Who is doing the thinking? How AI use shapes ownership and skill

AI raises a related question: when a tool does part of the work, who builds the skill? In the British Journal of Educational Technology, Yanyi Wu and Xinyu Lu studied GenAI reliance and agency among 342 university students in 85 groups in an AI-supported collaborative writing course, measured three times. When a student leaned on generative AI more than usual, their sense of agency, their ownership of the work, tended to drop afterward. It is observational and involved students, not employees, so we translate with care. The authors advise keeping the learner's responsibility for interpretation and judgment visible.

A second study, on a multi-agent AI system, shows setup matters too. Xiaoyan Chu, Hongzhu Dai, and Xuesong Zhai worked with 87 students in an instructional design course using the system. Groups working with the agents together were linked to higher design project scores, while individuals working alone showed bigger gains in critical thinking awareness. One course and a quasi-experiment make it an early signal. Together, the two studies point the same way: how much people lean on AI, and how it is configured, shapes what they get better at.

Our view: AI should serve the learning goal, not set it. As the episode puts it, "So the question isn't, should we use AI? It's, what do we want people to get better at?" If an AI assistant writes the action plan, whose plan is it?

Overloaded sessions: what you cut matters

In the International Journal of Training and Development, Nattapon Aunhabundit, I-Chun Chen, and Allan Lu studied graphic recording with 100 managers at one company. Half had training with graphic recording, an artist drawing key ideas live. The more overloaded people felt, the worse their reactions and test scores, but that drop was flatter in the graphic recording group. Think of it as a cushion, not a cure. The bigger fix starts with what you cut. The study measured reactions and a knowledge test right after training, not on-the-job behavior. Whether people use it depends on what comes next.

What shapes whether new skills stick after training

Transfer, using what you learned once you are back on the job, is where most training succeeds or fails. Two studies in the European Journal of Training and Development show how much depends on what follows.

Joerg Hruby and Carlos Canfield surveyed 138 alumni of global leadership programs six to twelve months after training. Global mindset and transfer climate, meaning the support around people back at work, both predicted transfer, and where the climate was strong, individual capability mattered less. The authors suggest tailoring post-training support to each learner. The design is cross-sectional and self-reported, so it shows association, not cause.

Charlotte Diekmann and Alexander Pundt tracked 62 leaders after leadership training, checking in every week for five weeks. The link between average transfer satisfaction and burnout tendency depended on how much that satisfaction swung from week to week. Their title calls transfer an emotional rollercoaster, and many programs step away right when the hard part begins. Support on the job belongs inside the design.

Three takeaways

  1. Measure what people can do, not who attended. As employers lean on in-house development, completion rates are not evidence of capability. Check that your measures are built to measure what you claim.
  2. Design AI use so human judgment stays in the lead. Heavier reliance on AI was linked to lower ownership, and how AI was configured shaped what learners gained. Decide what people must be able to do first, then where AI fits.
  3. Plan for the stretch after the course ends. Transfer climate predicted whether skills were used, and transfer satisfaction moved week to week. Support on the job is part of the design.

The practical challenge

Pick one course your team developed this year and ask: how do we know it changed what people do? If the answer is completion rates, that is your starting point.

Let's design for what people must be able to do

AI can build training in an afternoon now. The hard part is making sure it works on the job. That is what our Complete Learning Architecture approach is for: defining what people need to do differently, the support around them at work, where AI helps and where human judgment must lead, and how you will know it worked.

Our work is led by a United States Olympic and Paralympic Committee Coach Educator of the Year with 30+ years of experience in learning and development. If your next initiative needs to change what people do, we invite you to consider Emerge ID for your next project: https://emergeid.com/

Find more episodes of L&D Insights at Altitude every two weeks.

Sources cited in Episode 1

This image shows the author of the blog posts.

Written by Chris Packert

Chris Packert is the founder and CEO of Emerge ID, a learning architecture and instructional design company that helps organizations design, build, launch, and improve meaningful learning experiences. With more than 30 years of experience across K–12 education, higher education, workforce development, military distance education, and sport education, Chris brings a practical, systems-based approach to learning strategy. His work focuses on helping teams clarify performance needs, create accessible and engaging learning solutions, implement effective learning technology, and measure impact beyond completion data. Through Emerge ID, Chris partners with organizations to turn complex learning challenges into clear, usable, and measurable learning experiences.

More Posts by Chris

Related Posts

0 Comments