Since I went freelance in 2010 most of my work in European projects has been on the issue of monitoring quality of the online course that we are developing. So I thought I would wrap up this tour of online learning with a few observations on how this has been done.
Tracking quality should be an iterative process following a ConCurrent Design, Action Research or Design Thinking Cycle. Therefore what is described in linear terms below is actually a repeating cycle in order to accommodate what has been learned and implemented as a result of the monitoring.
Starting points
The monitoring of the course development and piloting has to take the project aims as its starting point. The aims will let you know what to monitor.
In the dCCD FLITE project for example the starting points were:
- Usability of the course
- Structured self-directed learning
- Group processes, such as Distributed ConCurrent Design (dCCD), to carry out a project
- Raising entrepreneurial skills/awareness
- Facilitating knowledge transfer
- Cross-sectorial benefits
Sources of information
It is especially important at the time of developing an online course to triangulate the feedback you are getting by making sure that you have several different sources of feedback on the issues that were agreed as important at the outset.
In the dCCD FLITE project our sources of feedback included
- VLE statistics
- Learners
- Course tutors
- Quality Board
- The final product of the course (in this case a business plan)
- Learners’ line manager or tutor
Not all the sources can give feedback on all the issues. For example, it is not really possible to look at the VLE statistics to gauge level of entrepreneurial awareness. However, the ideal situation would be that at least three of your sources can give feedback on each of your important factors. This ensures triangulation ie that you don’t get biased feedback from only one source, such as a tutor who does not want to acknowledge problems.
On a new course it is useful to have an independent board of interested external partners who can be relied on to give you unbiased feedback. This is why we have a Quality Board of about 15 volunteers giving feedback at various points in the course development and piloting.
Feedback tools and procedures
- Usability tests
- Statistics
- Quantitative surveys
- Qualitative surveys
Usability: It is important to check that the way your online course is presented is intuitive and easy to use for your prospective learners. A quick and effective way of doing this is to follow Steve Krug’s methodology and get a handful of people not connected with the project to carry out a few simple tasks related to the course such as, logging on, navigating to a specific place, editing a profile and adding a forum contribution. See a demo here:
Statistics: Most VLEs will give you a ton of statistics which you can use to track who logged on, how often, how long they stayed, what they did and what they posted. This is a powerful argument for using a VLE rather than some of the more free-range options such as a WordPress blog, as a home for your course. Though convergence is a strong trend and a WordPress blog is no longer just a blog and can have a great deal of additional functionality such as activity tracking.
Quantitative surveys: We are bombarded with surveys and most people will spend very little time and thought on completing your survey. Nevertheless they can be useful sources of feedback especially if undertaken soon after the event. Giving people multiple choice or true-false questions helps to speed up the process. You should also make sure the survey is not so long that too many refuse to complete it.
Qualitative surveys: These take more time and are based on either free text surveys or interviews. My experience has been that although these are more time-consuming they tend to result in the most useful feedback. I observe that the automatic polls tend to give a rosy picture while these qualitative interviews usually give you lots more information about where the problems are.
Longterm surveys: Especially in the case of skill acquisition, the true value of a course is often not apparent until a while has elapsed after doing the course.
Using the data
In the design cycle, this is the stage at which to adapt the original product, in our case the online course, in response to the information generated by the data. This stage is not so simple as often the data conflicts but hopefully the main trends are apparent.