On Wednesday, September 24, 2014, we hosted a webinar called “Learn how the National Health Service UK is using the Experience API”. As usual, the attendees had more questions that we could answer during the live webinar, so we’ve posted the questions and the answers here.
Q: You know what would be helpful is a matrix of different content types and the methodology used to equip them for Experience API.
A: I agree. We are hoping to increase the robustness of our xAPI Adopters page in the future to more fully describe the xAPI enabled tools on the market. From a technical perspective, we hope that xAPI recipies will help inform how to best outfit different content types.
Q: So I know how to add Experience API to a formally produced eLearning lesson, but not how to do it if somebody watches a video or reads a book.
Q: How do you track off-line activities such as workshops to incorporate into the LRS to evaluate the learning curve?
A: The technology to capture non-traditional learning events is still being developed and I expect it will be an area in which we see a lot of innovation over the next few years. We have some examples posted of how you could track videos watched through YouTube. We also published a simple prototype mobile book scanner app that allowed you to scan the UPC code on a book and record that you read it. One can easily imagine a classroom attendance tracking application that makes xAPI statements. These are just simple examples but hopefully they give you an idea of how these things could be done.
Q: Where is the SCORM content hosted in this set up? Inside of Watershed or some other server?
A: The SCORM Content is being hosted in SCORM Cloud, but launched via any of the existing NHS LMS’s, such as Oracle LMS or Moodle.
Q: What was the biggest thing you got wrong during the project and how did you tackle that?
A: From Richard – For me it was underestimating the amount of resources required to run the pilots. As I said in the webinar, this has been a very challenging project and I think we all underestimated just how much time it would consume (we’ve both worked weekends and evening to make it a success). Having said that, I think once the technology matures, projects like this will become easier and less time consuming. If someone approached me today to produce a piece of SCORM elearning, it would be easy using the available software, but 10 years ago this would be really challenging. I hope that one day, the technology will become second nature and easy to implement.
Q: What LMSs has your LRS been successfully integrated with?
A: Rustici Software actually offers three different LRS products:
- SCORM Engine – includes a xAPI LRS that is designed to be embedded in an LMS so that the LMS vendor may offer xAPI functionality. The SCORM Engine LRS powers the xAPI implementations behind Blackboard, Litmos and many other LMSs.
- SCORM Cloud – includes a free LRS that implements the basic xAPI functionality. This LRS stands alone is mostly used by people wanting to try out xAPI.
- Watershed LRS – our flagship standalone LRS designed to be a powerful tool for organizations to use. Watershed is the tool that adds sophisticated data analysis tools on top of the core xAPI implementation. The integration points between an organizational LRS and an LRS are complicated so we should probably chat if you want to really understand how this works. There are no out of the box Watershed/LMS integrations right now, but we’ve always been able to get the data to the right places for customers.
Q: What is the difference between Watershed and xAPI?
A: Watershed is the Learning Record Store (LRS). xAPI is the communication protocol between the statement generating apps and the LRS.
Q: What are some of the hard questions?
A: Richard – Most of the questions have been pretty tough! I like to think of ourselves as intrepid explorers like Columbus or Cook, sailing into uncharted territory, mapping the digital unknown as we progress! That means we have been challenged at every stage as nobody has done any of this before. I think the hardest questions for me have not been deciding how to collect the data or what to do with it, but deciding how we are going to limit ourselves and which items to remove from the scope of the project so we succeed.
Q: We have been using xAPI 0.95 (we wrote our own LRS) but now that we got loads of data in it, we are having a hard time to correlate that data with business data to answer some questions. Technically, how is NHS extracting valuable information from LRS?
A: First, thank you for being an early adopter. Please consider upgrading your system to the 1.0 version of xAPI, prior versions are considered drafts and shouldn’t be in production for long. The sooner we can get all of the early adopters up to 1.0 the better off the entire industry will be. To answer your question, our view is that it is the job of the LRS to provide the data structuring and analytics tools necessary to make meaning out of the xAPI data (it’s not an easy task). NHS is using the native Watershed functionality to get at its data.
Q: Do you know of companies that are offering LRS’s to people who would like to start recording their personal experiences?
A: You can use the free LRS provided via SCORM Cloud to track your own experiences. It’s not a fully featured and robust “personal data locker (PDL)”, but it can do the job. I expect it will be a few years before fully viable PDLs emerge in the mass market.
Q: Do you know of other software vendors who are embracing the xAPI?
Q: Authoring tools that support the xAPI?
Q: Our LMS Solution Provider (SuccessFactors) is very reluctant to commit to a formal position on support of the xAPI in the future. Do you know of current LMS Solutions who HAVE committed to supporting xAPI?
A: We publicize all of the adopters that we are aware of on our xAPI Adopters page.
Q: To Richard –> How does negative correlation be good for the Confidence x Competence assessment
A: From Richard: The intention is the user who completes the learning moves from “unconsciously incompetent” to “consciously incompetent”. It may sound like this is the wrong way round but what we are expecting to see is users self-assess higher at the start of the learning intervention as they are unaware of their failings. As the learner moves through the learning, we expect them to become more conscious of their behaviour so score themselves lower at the end of the intervention. Behaviour change in action!
Alison’s initial research results certainly shows this correlation as learners become more aware of their own behaviours.
Q: Is there a timeline to stop supporting the old SCORM?
A: I suspect SCORM will be with us for a long time to come. I like to use the analogy of VHS tapes. DVDs have been around for almost 20 years, but I only recently got rid of my VCR. SCORM is likely to follow a similar path. There is a massive investment in SCORM that won’t go away overnight. LMS vendors will continue to offer SCORM support for many years. I suspect you will start to see new tools that are xAPI only over the next few years, but the legacy tools that work with SCORM will continue to do so for a while.
Q: Any examples in the adult education sector?
Q: Prior learning and assessment for recognition of experiential learning in a formal academic system?
A: These are great use cases, unfortunately I don’t have any real world examples to share just yet.
Q: SCORM was designed to help with different LMS issues (not perfect but it was). How did that relate?
A: SCORM really only helps with the interoperability between elearning content and the LMS for delivery and tracking purposes. SCORM doesn’t help much with the consolidation of data from disparate LMSs.
Q: The idea was design a course using any tool and export to SCORM 1.2 and it would work (testing needed). I was never an evangelist of SCORM (just a developer that had to export it) and I hope xAPI can deliver.
A: Me too!
Q: One of the things that I saw early on in xAPI in various blogs was security issues. Is this still an issue?
A: (discussed on the webinar) Most of the discussion centers around privacy as opposed to security. (As far as we know) there aren’t any inherent security issues with xAPI, in fact xAPI was designed to significantly improve upon the inherently insecure SCORM model. Now, an LRS will be storing a lot of data, so it will be incumbent upon those system developers to use robust security practices to protect that data, but that is an issue with any software development. Those same developers will also need to take reasonable precautions to protect learners’ privacy and not share data inappropriately.
Q: I’d like to hear more about using xAPI data to do level 3 and 4 assessments. Maybe topic for a future webinar?
A: Absolutely, there’s some excited stuff happening here. xAPI allows you to capture both behavior and performance data in the same format as our learning data. Register for the PANDORA webinar to hear how they are measuring sales KPI data and looking for correlations with training programs.
Q: Is the ‘big apple’ system you refer to the ELFH (eLearning for Healthcare system) which is free to NHS, contains about 6k courses?
A: The ‘Big Apple’ refers to the Oracle Learning Management (OLM) system which is part of the national Electronic Staff Record (ESR) system. It is available to 100% of National Health Service employees but many organisations choose to promote alternative LMS’s, hence the 55% adoption. The eLearning for Healthcare system is a separate LMS, although shares many of the same nationally available courses as OLM.
Q: How many people did the pilots involve?
A: We are operating two distinct pilots under the umbrella of the National Health Service. The south of England pilot is running in a hospital across 3 acute wards, totalling around 60 users. The north of England pilot is running in an ambulance service across a subset of 100-200 users, although potentially could involve all 5000 employees of the organisation in the future.
Q: Is competency self-assessed, or is there an objective measure?
Q: Is all competency being only self-assessed?
A: For the pilot, these are self-assessed. We did discuss peer-assessments or manager-assessments but this is an example of one of the ideas we had to restrict due to the need to narrow the scope of the project.
Q: Is actor merging just a query, or does data get updated?
A: The xAPI statements themselves never change. The LRS system tracks the merged actors.
Q: If you don’t prescribe learning, how do you achieve compliance? Or “show” compliance…
A: In this case, the learning isn’t related to compliance.
From Richard – The bigger question is how you demonstrate compliance. If it’s a “tick-box” exercise, then this approach wouldn’t work, but if you want to demonstrate understanding then this is arguably more effective than a simple yes or no to the person completing an eLearning course.
Q: How would you be able to tell if it was the method or the content which led to it being seen as less effective?
A: Good question. As start to have more visibility into data, we often find that it just generates more questions and we need further experimentation to peel back the layers of the onion.
Q: Why is it called xAPI?
A: The original research project was called Project xAPI, and the name stuck. The name is meant to invoke tying two xAPIs together with a string to form a rudimentary telephone. In other words, a very simple way to communicate.
Q: How many LMSs or what percentage adopted/supported xAPI nowadays?
A: It’s hard to say exactly, but you can find a list here: ../adopters/
Q: How integrated is Watershed LRS as far as LCMS capabilities tied in xAPI?
A: Watershed can be easily connected with any application that supports the Experience API. Watershed very intentionally does not have any of its own LCMS capabilities. It is not trying to be a full featured LMS or LCMS, but rather an entirely new class of software.
Q: How do you verify the information an individual chooses to store?
Q: Can you clarify how companies are verifying data entered by an individual?
A: Good question. I don’t know that you ever can completely verify self reported information.
Q: For the SCORM Cloud where are the physical e-learning files located?
A: On the SCORM Cloud servers hosted on the Amazon infrastructure.
Q: Did NHS use Watershed LRS to track the data or use their Own LRS?
A: They used Watershed LRS
Q: What do you mean by tracking real world activities? Can you please explain?
A: Learning doesn’t just happen on a computer, it happens in classrooms and in conversations. It happens when reading a book or practicing a task. xAPI allows us to capture and represent these activities.
Q: Where was the Watershed LRS hosted and by who?
A: Watershed LRS is a SaaS product hosted by Rustici Software on the Amazon cloud infrastructure.
Q: How are you authenticating users from mobile devices when attaching to the LRS?
A: Users logging into Watershed LRS use HTTPS Basic Authentication. External mobile systems posting statements into the LRS would have their own distinct keys, again over HTTPS Basic Authentication.
Q: As an independent e-Learning Designer and Developer, I embrace the xAPI concept but I struggle with how to get my clients ‘on board’ … any suggestions?
A: (discussed on the webinar) Look for opportunities to start small. Find specific projects that can benefit from the introduction of a new capability and use that as the launching point for xAPI. Use the opportunity of extra data collection to introduce an analytical mindset into the training and learning department. Don’t try to do it all at once, it’s much easier to convince somebody to pilot a system than it is to try to get them to completely revamp an enterprise architecture.
Q: Are there any military users yet?
A: I think the military implementations are mostly in their early stages…but they’re coming!
Q: We have an SAP LMS. SAP say that it cannot host an LRS. Is this correct?
A: “Can’t” or “Won’t” or “Doesn’t have yet”? I don’t imagine that they actually “can’t” do it, they have plenty of smart developers to make that happen. They probably mean that they have not yet introduced the capability, which is a correct statement. If you would like to start using xAPI before your enterprise LMS is ready for it, we have solutions around Watershed LRS that can help. The great part about xAPI is that it is inherently open and natively supports the synchronization of data between systems. That means that you can start using Watershed to take advantage of xAPI today, but know that once your LMS catches up, all of that data is easily ported into the system of record.
Q: Are the badges similar to certifications, professional and nonprofessional?
A: Yes! Conceptually they are the same thing.
Q: Do all the learning statements reach the LRS via SCORM Cloud?..apart from the bookmarklet statements?
A: No. The only statements being generated by SCORM Cloud are from the e-Learning courses.
Q: Are the 2 regions sharing a badge server? Or separate servers?
A: Both are using a single instance of Watershed LRS. Watershed is hosted across many servers.
Q: In regard to NHS and Watershed, what did the consulting relationship look like? How long did it take to plan and implement the pilot? What was the cost? Would love to see a follow up to this after the results come in.
A: The project kicked off in February. Immediately, we started into the conceptual work, determining both the contributing activities and the potential visualizations that would be presented in Watershed. Working on these in concert allows us to be sure we’re designing things that are possible to collect and present.
March and April included several iterations on the visualization, and some troubleshooting on some of the more challenging activity providers. Because this project was early in the Watershed First lifecycle, many of the steps took longer than they would if the project were starting today.
May and June were testing months, and came with bug fixes and certain changes. And by July and August, the project was fully underway. In total, about 6 months of calendar time from start to finish.
Q: What level of training is required for administrators to work competently with xAPI to produce comprehensive reports on an organization’s activity.
A: Working directly with the Experience API is work for a software developer. An administrator is more likely to use a tool like Watershed LRS to produce reports. Watershed offers two types of reports, user-configurable and custom. The user-configurable reports are quite simple to set up. The custom reports require Rustici Software’s help.
Q: What does it take to prototype this technology using a real-life scenario? We have a project in-house that could benefit from this technology. Is there an opportunity to build something as a demo for some of our internal stakeholders?
A: Absolutely. The scope depends on what you want to do. If you’re using existing tools that are already outfitted with xAPI we can set up a demo in a matter of minutes. Other situations become more complicated but we can talk you through it.
Q: Will you be sending out the slides from today’s presentation?
Q: will we be able to get a recording of this webinar?
Q: Will this be recorded for access later?
Q: will a recording of this presentation be available for later viewing?
Q: Will a recording of the Webinar be delivered to attendees after the event? Thanks
Q: Was this session recorded for future playback? I have others in our company who could benefit from this session.
Q: Slides available?
Q: Is this slide deck available?
Q: I need to leave halfway through, just checking if you are recording the webinar?
Q: do we get a handout or something like that?
Q: can we have a copy of this webinair
Q: Will this seminar be available for later viewing?
Q: Can you send me a link to the recording of this session? There are some other folks at my organization that would like to watch and were not able to attend the session today.
A: Yes, we always make our webinar recordings and slides available. You can download them here.
Q: Great session. I will definitely participate in future offerings.
Q: Thanks to Alison and Richard for an excellent ‘double act’ presentation 🙂
Q: Thanks for sharing.
Q: Thanks a million. It’s nice to see this come to life. Is a familiar method to what i’m trying to implement for talent tracking at my job.
Q: Thank you very much!
Q: Thanks, folks. Very informative.
Q: LOVE this! Thank you!!!!!!!!!
Q: Great presentation. Very Helpful!
Q: Absolutely inspiring, thanks everyone!
Q: Very informative and exciting to see xAPI being used in the health care industry.
Q: Very good presentation.
Q: Very cool! I LOVE this!!!!
Q: Thanks for sharing
Q: Thank you for the presentation
Q: Keep on the good work!! 🙂
Q: Great work NHS UK!! Keep it up in paving a new and fresh way!!
Q: Great webinar. Very informative and helpful to see real use cases of xAPI in action.
Q: Extremely exciting stuff, thanks again!
Q: Excellent webinar! I don’t think my organization is ready for an LRS, but just listening to the issues that the presenters from NHS had to deal with was fascinating. Q: Thanks.
Q: Thanks!
Q: Very appropriate and timely discussion. Our organization is facing this very question right now. I’m so glad our UK friends were able to share first. I’d be interested to hear more about the setup/startup conversation as part of the planning and fine-tuning that had to go into the final result. We need more forward-thinking webinars like this that show the blending of theory and practice.
Q: Looking forward to seeing xAPI take off!
Q: I loved the multiple, loosely connected activity stuff.
A: Glad you liked it! Thanks for the compliments and you’re welcome!
