{"id":826,"date":"2014-11-16T06:45:45","date_gmt":"2014-11-16T14:45:45","guid":{"rendered":"http:\/\/opentextbc.ca\/teachinginadigitalage\/?post_type=chapter&#038;p=826"},"modified":"2018-11-10T12:07:55","modified_gmt":"2018-11-10T20:07:55","slug":"section-7-5-strengths-and-weaknesses-of-moocs","status":"publish","type":"chapter","link":"https:\/\/opentextbc.ca\/teachinginadigitalage\/chapter\/section-7-5-strengths-and-weaknesses-of-moocs\/","title":{"raw":"5.4 Strengths and weaknesses of MOOCs","rendered":"5.4 Strengths and weaknesses of MOOCs"},"content":{"raw":"<blockquote>\r\n<p style=\"text-align: right\"><em>In-depth analysis by standard academic criteria shows that MOOCs have more academic rigor and are a far more effective teaching methodology than in-house teaching<\/em><\/p>\r\n<\/blockquote>\r\n<p style=\"text-align: right\">Benton R. Groves, Ph.D. student<\/p>\r\n\r\n<blockquote>\r\n<p style=\"text-align: right\"><em>My big concern with xMOOCs is their limitation, as currently designed, for developing the higher order intellectual skills needed in a digital world.<\/em><\/p>\r\n<\/blockquote>\r\n<p style=\"text-align: right\">Tony Bates<\/p>\r\n\r\n<h2>5.4.1 The\u00a0research on MOOCs<\/h2>\r\nBecause at the time of writing most MOOCs are less than four\u00a0years old, there are relatively few\u00a0research publications on\u00a0MOOCs, although research activities are now beginning to pick up. Much of the research to date on MOOCs comes from the institutions offering MOOCs, mainly in the form of reports on enrolments, or self-evaluation by instructors. The commercial platform providers such as Coursera and Udacity have provided limited research information\u00a0overall, which is a pity, because they have access to really big data sets. However, MIT and Harvard, the founding partners in edX, are conducting some research, mainly on their\u00a0own courses. There is very little independent research to date on either xMOOCs or cMOOCs.\r\n\r\nHowever, wherever possible, I have tried to use any\u00a0research that has been done that provides insight into the strengths and weaknesses of MOOCs. At the same time, we should be clear that we are discussing a phenomenon that to date has been marked largely by political, emotional and often irrational discourse, and in terms of cumulative hard evidence, we will have to wait for some time.\r\n\r\nLastly, it should be remembered when I am evaluating MOOCs I am applying the criteria of whether MOOCs are likely to lead to the kinds of learning needed in a digital age: in other words, do they help develop the knowledge and skills defined in <a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/chapter\/section-1-3-the-skills-needed-in-a-digital-age\/\">Chapter 1<\/a>?\r\n<h2>5.4.2 Open and free education<\/h2>\r\nMOOCs, particularly xMOOCs, deliver high quality content from some of the world's best universities for free to anyone with a computer and an Internet connection. This in itself is an amazing value proposition. In this sense, MOOCs are an incredibly valuable addition to educational provision.\u00a0Who could argue against this? Certainly not me, so long as the argument for MOOCs goes no further.\r\n\r\nHowever, this is not the only form of open and free education. Libraries, open textbooks and educational broadcasting are also open and free and have been for some time, even if they do not have the same power and reach as Internet-based delivery. There are also lessons we can learn from these earlier\u00a0forms of open and free education that still\u00a0apply to MOOCs.\r\n\r\nThe first is that these earlier forms of open and free did not replace the need for formal, credit-based education, but were used to supplement or strengthen it. In other words, MOOCs\u00a0are a tool for continuing and informal education, which has high value in its own right. As we shall see though they work best when people are already reasonably well educated.\r\n\r\nThe problem comes when it is argued that because MOOCs are open and free to end-users, they will inevitably force down the cost of conventional higher education, or eliminate the need for it altogether, especially in developing\u00a0countries (see the Friedman comment at the beginning of this chapter.)\r\n\r\nThere have been many attempts in the past to use educational broadcasting and satellite broadcasting in developing\u00a0countries (see Bates, 1985), and they all failed substantially\u00a0to increase access or reduce cost\u00a0for a variety of reasons, the most important being:\r\n<ul>\r\n \t<li>the high cost of ground equipment (including security from theft or damage);<\/li>\r\n \t<li>the need for local support for learners without high levels of education, and the\u00a0high cost of local, 'ground' support;<\/li>\r\n \t<li>the need to adapt to the culture of the receiving countries;<\/li>\r\n \t<li>the difficulty of covering the operational costs of management and administration, especially for assessment, qualifications and local accreditation.<\/li>\r\n<\/ul>\r\nAlso the priority in most developing\u00a0countries is not for courses from high-level Stanford University professors, but for\u00a0programs for high schools. Finally,\u00a0although<span>\u00a0mobile phones are widespread in Africa, they operate on very narrow bandwidths. For instance, it costs US$2 to download a typical YouTube video \u2013 equivalent to a day\u2019s salary for many Africans. Streamed video lectures then have limited applicability.<\/span>\r\n\r\nThis is not to say that MOOCs could not be valuable in developing\u00a0countries, but\u00a0this will mean:\r\n<ul>\r\n \t<li>being realistic as to what they\u00a0can actually deliver;<\/li>\r\n \t<li>working in partnership with educational institutions and systems and other partners in developing countries;<\/li>\r\n \t<li>ensuring that the necessary local\u00a0support - which costs real money - is put in place;<\/li>\r\n \t<li>adapting the design, content and delivery of MOOCs to the cultural and economic requirements of\u00a0those countries.<\/li>\r\n<\/ul>\r\nFurthermore, MOOCs are not always open as in the sense of open educational resources. Coursera and Udacity for instance offer limited access to their material for re-use without permission. On other more open platforms, such as edX, individual faculty or institutions may restrict re-use of material. Lastly, many MOOCs exist for only one or two years then disappear, which limits their use as open educational resources for re-use in other courses or programs.\r\n\r\nFinally, although MOOCs are in the main free for participants, they are not without substantial cost to MOOC providers, an issue that will be discussed in more detail in Section 5.4.8.\r\n<h2>5.4.3 The audience that MOOCs mainly serve<\/h2>\r\nIn <a href=\"http:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=2381263\">a\u00a0research report<\/a> from Ho et al. (2014), researchers at Harvard University and MIT found that on the first 17 MOOCs offered through edX,\u00a0<span>66\u00a0per cent\u00a0of all participants, and 74\u00a0per cent\u00a0of all who obtained a certificate, have a bachelor\u2019s degree or above, 71 per cent were male, and the average age was 26. This and other studies also found that a high proportion of\u00a0participants\u00a0came from outside the USA, ranging from 40-60\u00a0per cent\u00a0of all participants, indicating strong interest\u00a0internationally in\u00a0open access to high quality university teaching.<\/span>\r\n\r\nIn a study\u00a0based on over 80 interviews in 62 institutions 'active in the MOOC space', <a href=\"http:\/\/cbcse.org\/wordpress\/wp-content\/uploads\/2014\/05\/MOOCs_Expectations_and_Reality.pdf\">Hollands and Tirthali (2014)<\/a>, researchers at Columbia University Teachers' College, found that:\r\n<blockquote><em>Data from MOOC platforms indicate that MOOCs are providing educational opportunities to millions of individuals across the world. However, most MOOC participants are already well-educated and employed, and only a small fraction of them fully engages with the courses. Overall, the evidence suggests that MOOCs are currently falling far short of \u201cdemocratizing\u201d education and may, for now, be doing more to increase gaps in access to education than to diminish them.<\/em><\/blockquote>\r\nThus MOOCs, as is common with most forms of university continuing education, cater to the better educated, older and employed sectors of society.\r\n<h2>5.4.4 Persistence and commitment<\/h2>\r\nThe edX researchers (Ho et al., 2014)\u00a0identified different levels of commitment as follows across 17 edX MOOCs:\r\n<ul>\r\n \t<li><strong>only registered<\/strong>: registrants who never access the courseware (35\u00a0per cent);<\/li>\r\n \t<li><strong>only viewed<\/strong>: non-certified registrants who access the courseware, accessing less than half of the available chapters (56\u00a0per cent);<\/li>\r\n \t<li><strong>only explored<\/strong>: non-certified registrants who access more than half of the available chapters in the courseware, but did not get a certificate (4\u00a0per cent);<\/li>\r\n \t<li><strong>certified<\/strong>: registrants who earn a certificate in the course (5\u00a0per cent).<\/li>\r\n<\/ul>\r\nHill (2013) has identified five types of participants in Coursera courses:\r\n\r\n<a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg\"><img src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg\" alt=\"Print\" width=\"755\" height=\"26\" class=\"aligncenter size-full wp-image-1668\" \/><\/a>\r\n\r\n[caption id=\"attachment_784\" align=\"aligncenter\" width=\"746\"]<a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/Hills-MOOC-participants-2.jpg\"><img src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/Hills-MOOC-participants-2-1024x750.jpg\" alt=\"\u00a9 Phil Hill, 2013\" width=\"746\" height=\"548\" class=\" wp-image-784\" \/><\/a> Figure 5.4.4.1 \u00a9 Phil Hill, 2013[\/caption]\r\n\r\n<a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg\"><img src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg\" alt=\"Print\" width=\"755\" height=\"13\" class=\"aligncenter size-full wp-image-1580\" \/><\/a>\r\n\r\nEngle\u00a0(2014) found\u00a0similar patterns for the University of British Columbia MOOCs on Coursera (also replicated in other studies):\r\n<ul>\r\n \t<li>of those that initially sign up, between one third and a half do not participate in any other active way;<\/li>\r\n \t<li>of those that participate in at least one activity, between 5-10\u00a0per cent\u00a0go on to successfully complete a certificate.<\/li>\r\n<\/ul>\r\nThose going on to achieve certificates usually are within the 5-10\u00a0per cent\u00a0range of those that sign up and in the 10-20\u00a0per cent\u00a0range for those who actively engaged with the MOOC at least once. Nevertheless, the numbers obtaining certificates are still large in absolute terms: over 43,000 across 17 courses on\u00a0edX and 8,000 across four courses at UBC (between 2,000-2,500 certificates per course).\r\n\r\n<a href=\"http:\/\/jolt.merlot.org\/vol9no2\/milligan_0613.htm\">Milligan et al<\/a>. (2013) found a similar pattern of commitment in cMOOCs, from interviewing a small sample of participants (29 out of 2,300 registrants) about halfway through a\u00a0cMOOC:\r\n<ul>\r\n \t<li>passive participants: in Milligan's study these were those that felt lost in the MOOC and rarely but occasionally logged in;<\/li>\r\n \t<li>lurkers: they were actively following the course but did not engage in any of the activities (just under half those interviewed);<\/li>\r\n \t<li>active participants (again, just under half those interviewed) who were fully engaged in the course activities.<\/li>\r\n<\/ul>\r\n<span>MOOCs need to be judged for what they are, a somewhat unique \u2013 and valuable \u2013 form of non-formal education.\u00a0<\/span>These results are very similar to research into non-formal educational broadcasts (e.g. the History Channel). One would not expect a viewer to watch every episode of a History Channel series then take an exam at the end. Ho et al. (p.13) produced the following diagram to show the different levels of commitment to xMOOCs:\r\n\r\n<a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg\"><img src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg\" alt=\"Print\" width=\"465\" height=\"16\" class=\"aligncenter wp-image-1668\" \/><\/a>\r\n\r\n[caption id=\"attachment_780\" align=\"aligncenter\" width=\"457\"]<a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/edX-participants-2.jpg\"><img src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/edX-participants-2.jpg\" alt=\"Ho et al., 2014\" width=\"457\" height=\"587\" class=\" wp-image-780\" \/><\/a> Figure 5.4.4.2 Level of participation in MOOCs\u00a0 \u00a9\u00a0Ho et al., 2014[\/caption]\r\n\r\n<a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg\"><img src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg\" alt=\"Print\" width=\"465\" height=\"8\" class=\"aligncenter wp-image-1580\" \/><\/a>\r\n\r\nNow compare that to what I wrote in 1985 about educational broadcasting in Britain (Bates, 1985):\r\n<blockquote><span>(p.99): <em>At the centre of the onion is a small core of fully committed students who work through the whole course, and, where available, take an end-of-course assessment or examination. Around the small core will be a rather larger layer of students who do not take any examination but do enrol with a local class or correspondence school. There may be an even larger layer of\u00a0students who, as well as watching and listening, also buy the accompanying textbook, but who do not enrol in any courses. Then, by far the largest group, are those that just watch or listen to the programmes. Even within this last group, there will be considerable variations, from those who watch or listen fairly regularly, to those, again a much larger number, who watch or listen to just one programme.\u00a0<\/em><\/span><\/blockquote>\r\nI also wrote (p.100):\r\n<blockquote><em>A sceptic may say that the only ones who can be said to have learned effectively are the tiny minority that worked right through the course and successfully took the final assessment\u2026A counter argument would be that broadcasting can be considered successful if it merely attracts viewers or listeners who might otherwise have shown no interest in the topic; it is the numbers exposed to the material that matter\u2026the key issue then is whether broadcasting does attract to education those who would not otherwise have been interested, or merely provides yet another opportunity for those who are already well educated\u2026There is a good deal of evidence that it is still the better educated in Britain and Europe that make the most use of non-formal educational broadcasting.<\/em><\/blockquote>\r\n<span>Exactly the same could be said about MOOCs. In a digital age where easy and open access to new knowledge is critical for those working in knowledge-based industries, MOOCs will be one valuable source or means of accessing that knowledge. The issue is though whether there are more effective ways to do this.\u00a0<\/span>Thus MOOCs can be considered\u00a0a useful - but not really revolutionary -\u00a0contribution to non-formal continuing education.\r\n<h2>5.4.5 What do students learn in MOOCs?<\/h2>\r\nThis is a much more difficult question to answer, because so little of the research to date (2014) has tried to answer this question. (One reason, as we shall see in the next section, is that assessment of learning in MOOCs remains a major challenge). There are at least two kinds of study: quantitative\u00a0studies that seek to quantify learning gains; and qualitative studies that describe the experience of learners within MOOCs, which indirectly provide some insight into what they have learned.\r\n\r\nAt the time of writing, the most quantitative\u00a0study of learning in MOOCs has been\u00a0by <a href=\"http:\/\/www.irrodl.org\/index.php\/irrodl\/article\/view\/1902\/3009\">Colvin et al. (2014)<\/a>, who investigated 'conceptual learning' in an MIT Introductory Physics MOOC. They compared learner performance not only between different sub-categories of learners within the MOOC, such as those with no physics or math background with those such as physic teachers who had considerable prior knowledge, but also with on-campus students taking the same\u00a0curriculum in\u00a0a traditional campus teaching format. In essence, the study found no significant differences in learning gains between or within the two types of teaching, but it should be noted that the on-campus students were students who had failed an earlier version of the course and were retaking it.\r\n\r\nThis research is a classic example of the no significant difference in comparative studies in educational technology; other variables, such as differences in the types of students, were as\u00a0important as\u00a0the mode of delivery.\u00a0Also<span>, this MOOC design represents\u00a0a behaviourist-cognitivist approach to learning that places heavy emphasis on correct answers to conceptual questions. It doesn't attempt to\u00a0develop the skills needed in a digital age as identified in Chapter 1.<\/span>\r\n\r\nThere have been far more studies of the experience of learners within MOOCs, particularly focusing on the discussions within MOOCs (see for instance, Kop, 2011). In general (although there are exceptions), discussions are unmonitored, and it is left to participants to make connections and respond to other students comments. However<span>, there are some strong criticisms\u00a0of the effectiveness of the discussion element of MOOCs for\u00a0developing the high-level conceptual analysis\u00a0required for academic learning. To develop deep, conceptual learning, there is a need in most cases for intervention by a subject expert to clarify misunderstandings or misconceptions, to provide accurate feedback, \u00a0to ensure that the criteria for academic learning, such as use of evidence, clarity of argument, and so on, are being met, and to ensure\u00a0the necessary input and guidance to seek deeper understanding (see Harasim, 2013).<\/span>\r\n\r\n<span>Furthermore, the more massive the course, the more likely participants\u00a0are to feel \u2018overload, anxiety and a sense of loss\u2019, if there is not some instructor intervention or structure imposed (Knox, 2014). Firmin et\u00a0al. (2014) have shown that when there is some form of instructor \u2018encouragement and support of student effort and engagement\u2019, results improve for all participants in MOOCs. Without a structured role for subject experts,\u00a0participants are faced with a wide variety of quality in terms of comments and feedback from other participants.\u00a0There is again a great deal of research on the conditions necessary for the successful conduct of collaborative and co-operative group learning (see for instance, Dillenbourg, 1999, Lave and Wenger, 1991), and\u00a0these findings certainly have not been generally applied to the management of MOOC discussions to date.\u00a0<\/span>\r\n\r\n<span>One\u00a0counter argument is that at least cMOOCs develop a new form of learning based on networking and collaboration that is essentially different from academic learning, and MOOCs are thus\u00a0more appropriate to the needs of learners in a digital age. Adult participants in particular, it is claimed by Downes and Siemens, have the ability to self-manage the development of high level conceptual learning. \u00a0MOOCs are \u2018demand\u2019 driven, meeting the interests of individual students who seek out others with similar interests and the necessary expertise to support them in their learning, and for many this interest may well not include the need for deep, conceptual learning but more likely the appropriate applications of prior\u00a0knowledge in new or specific contexts.\u00a0<\/span>MOOCs do appear to work best for those who already have a high level of education and therefore bring many of the conceptual skills developed in formal education with them when they join a MOOC, and therefore contribute to helping those who come without such prior knowledge or skills.\r\n\r\n<span>Over time, as more experience is gained, MOOCs are likely to incorporate\u00a0and\u00a0adapt some of the findings from research on smaller group work to the\u00a0much larger numbers in MOOCs. For instance, some MOOCs are using 'volunteer' or community tutors (Dillenbourg, 2014). The US State Department has organized MOOC camps\u00a0through\u00a0<span>US missions and consulates abroad\u00a0<\/span>to mentor MOOC participants. The camps include Fulbright scholars and embassy staff who lead discussions on content and topics for MOOC participants in countries abroad (Haynie, 2014). Some MOOC providers, such as the University of British Columbia, pay a small cohort of academic\u00a0assistants to monitor and contribute to the MOOC discussion forums (Engle, 2014).\u00a0Engle reported that the<span>\u00a0use of academic assistants, as well as limited but effective interventions from the instructors themselves, made the UBC MOOCs more interactive and engaging. However, paying for people to monitor and support MOOCs will of course increase the cost to providers. Consequently, MOOCs are likely to develop\u00a0new automated ways to manage\u00a0discussion effectively in very large groups. The University of Edinburgh is experimenting with automated 'teacherbots' that crawl through online discussion forums and direct predetermined comments to students identified as needing help or encouragement\u00a0(Bayne, 2014).\u00a0<\/span><\/span>\r\n\r\n<span><span>These results and approaches are\u00a0consistent\u00a0with\u00a0prior research on the importance of instructor presence for successful for-credit online learning.\u00a0<\/span>In the meantime, though, there is much work still to be done if MOOCs are to provide the support and structure needed to ensure deep, conceptual learning where this does not already exist in students. The development of the skills needed in a digital age is likely to be an even greater challenge when dealing with massive numbers. However, we need much more research into what participants actually learn in MOOCs and under what conditions before any firm conclusions can be drawn.<\/span>\r\n<h2>5.4.6 Assessment<\/h2>\r\nAssessment of the massive numbers of participants in MOOCs has proved to be a major challenge. It is a complex topic that can be dealt with only briefly here. However, <a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/chapter\/5-8-assessment-of-learning\/\">Appendix 1, Section 8<\/a>\u00a0provides a\u00a0general analysis\u00a0of different types of assessment, and <a href=\"http:\/\/www.irrodl.org\/index.php\/irrodl\/article\/view\/1680\/2904\">Suen (2014)<\/a> provides a comprehensive and balanced overview of the way assessment has been used in MOOCs to date. This section draws heavily on Suen's paper.\r\n<h3 style=\"text-align: left\">5.4.6.1 Computer marked assignments<\/h3>\r\nAssessment to date in MOOCs has been primarily of two kinds. The first is based on quantitative multiple-choice tests, or response boxes where formulae or 'correct code' can be entered and automatically checked. Usually participants are given immediate automated feedback on their answers, ranging from simple right or wrong answers to more complex responses depending on the type of response checked, but in all cases, the process is usually fully automated.\r\n\r\nFor straight testing of facts, principles, formulae, equations and other forms of conceptual learning where there are clear, correct answers, this works well. In fact, multiple choice computer marked assignments were used by the UK Open University as long ago as the 1970s, although the means to give immediate online feedback were not available then. However, this method of assessment\u00a0is limited for testing deep or 'transformative' learning,\u00a0and particularly weak for assessing\u00a0the intellectual skills needed in a digital age, such as creative or original thinking.\r\n<h3 style=\"text-align: left\">5.4.6.2 Peer\u00a0assessment<\/h3>\r\nThe second type of assessment that has been tried in MOOCs has been peer assessment, where participants assess each other's work. Peer assessment is not new. It has been successfully used for formative assessment in traditional classrooms and in some online teaching for credit (Falchikov and Goldfinch, 2000;\u00a0van Zundert et al., 2010). More importantly, peer assessment is seen as a powerful way to improve deep understanding and knowledge through the process of students evaluating the work of others, and at the same time, it can be useful for developing some of the skills needed in a digital age, such as critical thinking, for those participants assessing other participants.\r\n\r\nHowever, a key feature of the successful use of peer assessment has been the close involvement of an instructor or teacher, in providing benchmarks, rubrics or criteria for assessment, and for monitoring and adjusting peer assessments to ensure consistency and a match with the benchmarks\u00a0set by the instructor. Although\u00a0an instructor can provide the benchmarks and rubrics in MOOCs,\u00a0close monitoring of the multiple peer assessments is difficult if not impossible with the very large numbers of participants. As a result, MOOC participants\u00a0often become incensed at being randomly assessed by other participants who\u00a0may not and often do not\u00a0have\u00a0the knowledge or ability to give a 'fair' or accurate assessment of a participant's work.\r\n\r\nVarious attempts to get round the limitations of peer assessment in MOOCs have been tried such as calibrated peer reviews, based on averaging all the peer ratings, and Bayesian post hoc stabilization (Piech at al. 2013), but although these statistical techniques reduce the error (or spread) of peer review somewhat they still do not remove the problems of systematic errors of judgement in raters due to misconceptions. This is particularly a problem where a majority of participants fail to understand key concepts in a\u00a0MOOC, in which case peer assessment becomes the blind leading the blind.\r\n<h3 style=\"text-align: left\"><strong>5.4.6.3 Automated essay scoring<\/strong><\/h3>\r\nThis is another area where there have been attempts to automate scoring (Balfour, 2013). Although such methods are increasingly sophisticated they are currently limited in terms of accurate assessment to measuring primarily technical writing skills, such as grammar, spelling and sentence construction. Once again they do not measure accurately essays where higher level intellectual skills are demonstrated.\r\n<h3 style=\"text-align: left\">5.4.6.4 Badges and certificates<\/h3>\r\nParticularly in xMOOCs, participants may be awarded a certificate or a 'badge' for successful completion of the MOOC, based on a final test (usually computer-marked) which measures the level of learning in a course.\r\n\r\nT<span>he American Council on Education (ACE), which\u00a0<span>represents the presidents of U.S. accredited, degree-granting institutions,<\/span>\u00a0recommended offering credit for five courses on the\u00a0Coursera MOOC platform.\u00a0<\/span>However, according to the person responsible for the review process (Book, 2013):\r\n<blockquote><em>what the ACE accreditation does is merely accredit courses from institutions that are already accredited.\u00a0The review process doesn\u2019t evaluate learning outcomes, but is a course content focused review thus obviating all the questions about effectiveness of the pedagogy in terms of learning outcomes.\u00a0<\/em><\/blockquote>\r\nIndeed, most of the institutions offering MOOCs will not accept their own certificates for admission or credit within their own, campus-based programs. Probably nothing says more about the confidence in the quality of the assessment than this failure of MOOC providers to recognize their own teaching.\r\n<h3 style=\"text-align: left\">5.4.6.5 The intent behind assessment<\/h3>\r\nTo evaluate assessment in MOOCs requires an examination of the intent behind assessment. There are many different purposes behind assessment (see <a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/chapter\/5-8-assessment-of-learning\/\">Appendix 1, Section 8<\/a>). Peer assessment and immediate feedback on computer-marked tests can be extremely valuable for formative assessment, enabling participants to see what they have understood and to help develop further their understanding of key concepts. In cMOOCs, as Suen points out, learning is measured as the communication that takes place between MOOC\u00a0participants, resulting in crowdsourced validation of knowledge - it's what the sum of all the participants come to believe to be true as a result of participating in the MOOC, so formal assessment is unnecessary. However, what is learned in this way\u00a0is not necessarily <em>academically<\/em> validated knowledge, which to be fair, is not the concern of cMOOC\u00a0proponents.\r\n\r\nAcademic assessment is a form of currency, related not only to measuring student achievement but also affecting student mobility (for example, entrance to graduate school) and perhaps more importantly employment opportunities and promotion. From a learner's perspective, the validity of the currency - the recognition and transferability of the qualification - is essential. To date, MOOCs have been unable to demonstrate that they are able to assess accurately the learning achievements of participants beyond comprehension and knowledge of ideas, principles and processes (recognizing that there is some value in this alone). What MOOCs\u00a0have not been able to demonstrate is that they can either develop or assess deep understanding or the intellectual skills required in a digital age. Indeed, this\u00a0may not be possible within the constraints of massiveness, which is their major distinguishing feature from other forms of online learning.\r\n<h2>5.4.7 Branding<\/h2>\r\nHollands and Tirthali (2014) in their survey on institutional expectations for MOOCs, found that building and maintaining brand was the second most important reason for institutions launching MOOCs (the most important was extending reach, which can also be seen as partly a branding exercise). Institutional branding through the use of MOOCs has\u00a0been helped by elite Ivy League universities such as Stanford, MIT and Harvard leading the charge, and by Coursera limiting access to its platform to only 'top tier' universities. This of course has led to a bandwagon effect, especially since many of the universities launching\u00a0MOOCs had previously disdained to move into credit-based online learning. MOOCs provided a way for these elite institutions to jump to the head of the queue in terms of status as\u00a0'innovators' of online learning, even though they arrived late to the party.\r\n\r\nIt obviously makes sense for institutions to use MOOCs to bring their areas of specialist expertise to a much wider public, such as the University of Alberta offering a MOOC on dinosaurs, MIT on electronics, and Harvard on Ancient Greek Heroes. MOOCs\u00a0certainly help to widen knowledge of\u00a0the quality of an individual professor (who is\u00a0usually delighted to reach more students in one MOOC than in a lifetime of on-campus teaching). MOOCs are also a good way to give a glimpse of the quality\u00a0of courses and programs offered by an institution.\r\n\r\nHowever, it is difficult to measure the real impact of MOOCs on branding. As Hollands and Tirthali put it:\r\n<blockquote><em>While many institutions have received significant media attention as a result of their MOOC activities, isolating and measuring impact of any new initiative on brand is a difficult exercise. Most institutions are only just beginning to think about how to capture and quantify branding-related benefits.<\/em><\/blockquote>\r\nIn particular,\u00a0these elite institutions do not need MOOCs to boost the number of applicants for their campus-based programs\u00a0(none to date is willing to accept successful completion of a MOOC for admission to credit programs), since\u00a0elite institutions have no difficulty in attracting already highly qualified students.\r\n\r\nFurthermore, once every other institution starts offering MOOCs, the branding effect gets lost to some extent. Indeed, exposing poor quality teaching or course planning to many thousands can have a negative impact on an institution's brand, as\u00a0Georgia Institute of Technology found when one of its MOOCs crashed and burned<a href=\"https:\/\/www.insidehighered.com\/news\/2013\/02\/04\/coursera-forced-call-mooc-amid-complaints-about-course\"> (Jaschik, 2013).<\/a>\u00a0However, by and large, most MOOCs succeed in the sense of bringing an institution's reputation in terms of knowledge and expertise\u00a0to many more people than\u00a0it would through\u00a0any other form of teaching or publicity.\r\n<h2>5.4.8 Costs and economies of scale<\/h2>\r\n<a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg\"><img src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg\" alt=\"Print\" width=\"755\" height=\"26\" class=\"aligncenter size-full wp-image-1668\" \/><\/a>\r\n\r\n[caption id=\"attachment_884\" align=\"aligncenter\" width=\"740\"]<a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/11\/Fixed-and-variable-costs-2.jpg\"><img src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/11\/Fixed-and-variable-costs-2.jpg\" alt=\"The MOOC value proposition is that MOOCs can eliminate the variable costs of course delivery. Image: \u00a9 OpenTuition.com, 2014 \" width=\"740\" height=\"393\" class=\" wp-image-884\" \/><\/a> Figure 5.4.8 The MOOC value proposition is that MOOCs can eliminate the variable costs of course delivery.\u00a0Image: \u00a9 OpenTuition.com, 2014[\/caption]\r\n\r\n<a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg\"><img src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg\" alt=\"Print\" width=\"755\" height=\"13\" class=\"aligncenter size-full wp-image-1580\" \/><\/a>\r\n\r\nOne main strength claimed for\u00a0MOOCs is that they are free to participants. Once again we shall see this is more true in principle than in practice, because\u00a0MOOC\u00a0providers may charge a range of fees, especially for assessment. Furthermore, although MOOCs may be free for participants, they are\u00a0not\u00a0without substantial cost to the provider institutions. Also, there are large differences in the costs of xMOOCs and cMOOCs, the latter being generally much cheaper to develop, although there are still some opportunity or actual costs even for cMOOCs.\r\n\r\nOnce again, there is very little information to date on the actual costs of designing and delivering a MOOC\u00a0as\u00a0\u00a0there are not\u00a0enough cases at the moment to draw firm conclusions about the costs of MOOCs. However\u00a0we do have some data. The <a href=\"http:\/\/www.uottawa.ca\/vr-etudes-academic\/en\/documents\/e-learning-working-group-report.pdf\">University of Ottawa<\/a>\u00a0(2013) estimated the cost of developing an xMOOC, based on figures provided to the university by Coursera, and on their own knowledge of the cost of developing online courses for credit, at around $100,000.\r\n\r\n<a href=\"http:\/\/flexible.learning.ubc.ca\/files\/2014\/09\/MOOC-Report.pdf\">Engle (2014)<\/a> has reported on the actual cost of five MOOCs from the University of British Columbia. (In essence, there were really four UBC MOOCs, as one was in two shorter parts.) There are two important features concerning the UBC MOOCs that do not necessarily apply to other MOOCs. First, the UBC MOOCs used a wide variety of video production methods, from full studio production to desktop recording, so development costs varied considerably, depending on the sophistication of the video production technique. Second, the UBC MOOCs made extensive use of paid academic assistants, who monitored discussions and adapted or changed\u00a0course materials as a result of student feedback, so there were substantial delivery costs as well.\r\n\r\n<span>Appendix B of the UBC report gives a pilot total of $217,657, but this excludes academic assistance or, perhaps the most significant cost, instructor time. Academic assistance came to 25\u00a0per cent\u00a0of the overall cost in the first year (<em>excluding<\/em> the cost of faculty). Working from the video\u00a0production costs ($95,350) and the proportion of costs (44\u00a0per cent) devoted to video\u00a0production in Figure 1 in the report, I estimate the direct cost at $216,700, or approximately $54,000\u00a0per MOOC,\u00a0<em>excluding<\/em> faculty time and co-ordination support (that is,\u00a0excluding program administration and overheads), but including academic assistance. However, the range of cost is almost as important. The video production costs for the MOOC which\u00a0used intensive studio production were more than six times\u00a0the video production costs of one of the other MOOCs.<\/span>\r\n\r\nThe main cost factors or variables in <em>credit-based<\/em> online and distance learning are relatively well understood, from previous research by Rumble (2001) and H\u00fclsmann\u00a0(2003). Using similar\u00a0costing methodology, I tracked and analysed the cost of an online master's program at the University of British Columbia over a seven year period (Bates and Sangr\u00e0, 2011). This program used mainly a learning management system as the core technology, with instructors both developing the course and providing online learner support and assessment, assisted where necessary by extra adjunct faculty for handling larger class enrolments.\r\n\r\nI found in my analysis of the costs of the UBC program that in 2003, development costs were approximately $20,000 to $25,000 per course. However, over a seven year period, course development\u00a0constituted less than 15\u00a0per cent\u00a0of the total cost, and occurred mainly in the first year or so of the program. Delivery costs, which included providing online learner support and student assessment, constituted more than a third of the total cost, and of course continued each year the course was offered. Thus in credit-based online learning, delivery costs tend to be more than double the development costs over the life of a program.\r\n\r\nThe main difference then between MOOCs, credit-based online teaching,\u00a0and campus-based teaching is that in principle\u00a0MOOCs eliminate all delivery\u00a0costs, because MOOCs do not provide learner support or instructor-delivered assessment, although again in practice this is not always true.\r\n\r\nThere is also clearly a large opportunity cost involved in offering xMOOCs. By definition, the most highly valued faculty are involved in offering MOOCs. In a large research university, such faculty are likely to have, at a maximum, a teaching load of four to six courses a year. Although most instructors volunteer to do MOOCs, their time is limited. Either it means dropping one credit course for at least one semester, equivalent to 25 per cent or more of their teaching load, or xMOOC development and delivery\u00a0replaces time spent doing research. Furthermore, unlike credit-based courses, which run from anywhere between five to seven years, MOOCs are often offered only once or twice.\r\n\r\nHowever one looks at it, the cost of xMOOC development, <em>without<\/em> including the time of the\u00a0MOOC instructor, tends to be almost double the cost of developing an online credit course using a learning management system, because of the use of video in MOOCs.\u00a0If the cost of the instructor is included, xMOOC production costs come closer to three times that of a similar length online credit course, especially given\u00a0the extra time faculty tend put in for such a public demonstration of their teaching in a MOOC. xMOOCs could (and some do)\u00a0use cheaper production methods, such as an LMS instead of video, for content delivery, or using and re-editing video recordings of classroom lectures via lecture capture.\r\n\r\nWithout learner support or academic\u00a0assistance, though, delivery costs for MOOCs are zero, and this is where the huge potential for savings exist. If the cost per participant is calculated the unit costs are very low. Even if the cost per student successfully obtaining an end of course certificate is calculated it will be many times lower than the cost of an online or campus-based successful student. If we take a MOOC costing roughly $100,000 to develop, and 5,000 participants\u00a0complete the end of course certificate, the average cost per\u00a0successful participant is $20. However, this assumes that the same type\u00a0of knowledge and skills is being assessed for both a MOOC and for a graduate masters program; usually\u00a0this not the case.\r\n\r\nThe\u00a0issue then is whether MOOCs can succeed without the cost of learner support and human assessment, or more likely, whether MOOCs can substantially reduce delivery costs through automation without loss of quality in learner performance. There is no evidence to date though that they can do this in terms of higher order learning skills and 'deep' knowledge. To assess this kind of learning requires setting assignments that test such knowledge, and such assessments\u00a0usually need human marking, which then adds to cost. We also\u00a0know from prior research from\u00a0successful online credit programs that active instructor online presence is a critical factor for successful online learning. Thus adequate learner support and assessment\u00a0remains a major challenge for MOOCs. MOOCs then are a good way to teach\u00a0certain levels of knowledge but will have major structural problems in teaching other types of knowledge. Unfortunately, it is the type of knowledge most needed in a digital world that MOOCs struggle to teach.\r\n\r\nIn terms of sustainable business models, the elite universities have been able to move into xMOOCs because of generous donations from private foundations and use of endowment funds, but these forms of funding are limited for most institutions. Coursera and Udacity have the opportunity to develop successful business models through various means, such as charging MOOC provider\u00a0institutions\u00a0for use of their platform, by collecting fees for badges or certificates, through the sale of participant data, through corporate sponsorship, or through direct advertising.\r\n\r\nHowever, particularly for publicly funded universities or colleges, most of these sources of income are not available or permitted, so it is hard to see how they can begin to recover the cost of a substantial\u00a0investment in MOOCs, even with 'cannibalising' MOOC material for on-campus use. Every time a MOOC is offered, this takes away resources that could be used for online credit programs. Thus institutions are faced with some hard decisions about where to invest their resources for online learning. The case for putting scarce resources\u00a0into MOOCs is far from clear, unless some way can be found to give credit for successful MOOC completion.\r\n<h2>5.4.9\u00a0Summary of strengths and weaknesses<\/h2>\r\nThe main points of this analysis of the strengths and weaknesses of MOOCs can be summarised as follows:\r\n<h3 style=\"text-align: left\">5.4.9.1 Strengths<\/h3>\r\n<ul>\r\n \t<li><span>MOOCs, particularly xMOOCs, deliver high quality content from some of the world's best universities for free to anyone with a computer and an Internet connection;<\/span><\/li>\r\n \t<li>MOOCs can be useful for opening access to high quality content, particularly in developing\u00a0countries, but to do so successfully will require a good deal of adaptation, and substantial investment in local support and partnerships;<\/li>\r\n \t<li>MOOCs are valuable for developing basic conceptual learning, and for creating large online communities of interest or practice;<\/li>\r\n \t<li>MOOCs are an extremely valuable form of lifelong learning and\u00a0continuing education;<\/li>\r\n \t<li>MOOCs have forced conventional and especially elite institutions to reappraise their strategies towards online and open learning;<\/li>\r\n \t<li>institutions have been able to extend their brand and status by making public their expertise and excellence in certain academic areas;<\/li>\r\n \t<li>MOOCs main value proposition is\u00a0to eliminate through computer automation and\/or peer-to-peer communication\u00a0the very large variable costs in higher education associated with providing learner support and quality assessment.<\/li>\r\n<\/ul>\r\n<h3 style=\"text-align: left\">5.4.9.2 Weaknesses<\/h3>\r\n<ul>\r\n \t<li>the high registration numbers for MOOCs are misleading; less than half of registrants actively participate, and of these, only a small proportion successfully complete the course; nevertheless, absolute numbers are still higher than for conventional courses;<\/li>\r\n \t<li>MOOCs are expensive to develop, and although commercial organisations offering MOOC platforms have opportunities for sustainable business models, it is difficult to see how publicly funded higher education institutions can develop sustainable business models for MOOCs;<\/li>\r\n \t<li>MOOCs tend to attract\u00a0those with already a high level of education, rather than widen access;<\/li>\r\n \t<li>MOOCs so far have been limited in the ability to develop high level academic learning, or the high level intellectual skills needed in a knowledge based society;<\/li>\r\n \t<li>assessment of the higher levels of learning remains a challenge for MOOCs, to the extent that most MOOC providers will not recognise their own MOOCs for credit;<\/li>\r\n \t<li>MOOC materials may be limited by copyright or time restrictions for re-use as open educational resources.<\/li>\r\n<\/ul>\r\n<div class=\"bcc-box bcc-info\">\r\n<h3 itemprop=\"educationalUse\">Activity 5.4\u00a0Assessing the strengths and weaknesses of MOOCs<\/h3>\r\n1. Do you agree that MOOCs are just another form of educational broadcasting? What are your reasons?\r\n\r\n<span>2. Is it reasonable to compare the costs of xMOOCs to the costs of online credit courses? Are they competing for the same funds, or are they categorically different in their funding source and goals? If so, how? <\/span>\r\n\r\n<span>3. Could you make the case that cMOOCs are a better value proposition than xMOOCs \u2013 or are they again too different to compare?\u00a0<\/span>\r\n\r\n4. MOOCs are clearly cheaper than either face-to-face or online credit courses if judged on the cost per participant successfully completing a course. Is this a fair comparison, and if not, why not?\r\n\r\n5. Do you think institutions should give credit for students successfully completing MOOCs? If so, why, and what are the implications?\r\n\r\n<\/div>\r\n<h1>References<\/h1>\r\n<span>Balfour, S. P. (2013)\u00a0<a href=\"Automated essay scoring and calibrated peer review. Research &amp; Practice in Assessment, 8.\" style=\"font-style: italic\">Assessing writing in MOOCs: Automated essay scoring and calibrated peer review<\/a>\u00a0<\/span><em>Research &amp; Practice in Assessment, <\/em>Vol. 8.\r\n\r\nBates, A. (1985) <em>Broadcasting in Education: An Evaluation<\/em> London: Constables\r\n\r\nBates, A. and Sangr\u00e0, A. (2011) <em>Managing Technology in Higher Education<\/em> San Francisco: Jossey-Bass\/John Wiley and Co\r\n\r\nBayne, S. (2014) <em>Teaching, Research and the More-than-Human in Digital Education<\/em>\u00a0Oxford UK:\u00a0EDEN Research Workshop (keynote: no printed record available)\r\n\r\nBook, P. (2103)\u00a0<em>ACE as Academic Credit Reviewer\u2013Adjustment, Accommodation, and Acceptance<\/em> <a href=\"http:\/\/wcetblog.wordpress.com\/2013\/07\/25\/ace-review-moocs-for-credit\/\">WCET Learn<\/a>, July 25\r\n\r\nColvin, K. et al. (2014) <em><a href=\"http:\/\/www.tonybates.ca\/2014\/10\/19\/what-students-learned-from-an-mit-physics-mooc\/#sthash.zqWeKuRA.dpuf\">Learning an Introductory Physics MOOC: All Cohorts Learn Equally, Including On-Campus Class<\/a><\/em>, IRRODL, Vol. 15, No. 4\r\n\r\nDillenbourg, P. (ed.) (1999)\u00a0<em>Collaborative-learning: Cognitive and Computational Approaches<\/em>. Oxford: Elsevier\r\n\r\nDillenbourg, P. (2014) <em>MOOCs: Two Years Later<\/em>, Oxford UK:\u00a0EDEN Research Workshop\u00a0<span>(keynote: no printed record available)<\/span>\r\n\r\nEngle, W. (2104) <a href=\"http:\/\/flexible.learning.ubc.ca\/files\/2014\/09\/MOOC-Report.pdf\"><em>UBC MOOC Pilot: Design and Delivery<\/em><\/a> Vancouver BC: University of British Columbia\r\n\r\nFalchikov, N. and Goldfinch, J. (2000) <em>Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks<\/em> <a href=\"http:\/\/rer.sagepub.com\/content\/70\/3\/287.short\">Review of Educational Research<\/a>, Vol. 70, No. 3\r\n\r\nFirmin, R. et al. (2014) Case study: using MOOCs for conventional college coursework\u00a0<em>Distance Education<\/em>, Vol. 35, No. 2\r\n\r\nHarasim, L. (2012)\u00a0<em>Learning Theory and Online Technologies<\/em>\u00a0New York\/London: Routledge\r\n\r\nHaynie, D. (2014). <a href=\"http:\/\/www.usnews.com\/education\/online-education\/articles\/2014\/01\/20\/state-department-hosts-mooc-camp-for-online-learners-abroad\">State Department hosts 'MOOC Camp' for online learners<\/a>. US News,January 20\r\n\r\nHill, P. (2013) <a href=\"http:\/\/mfeldstein.com\/validation-mooc-student-patterns-graphic\/\">Some validation of MOOC student patterns graphic<\/a>, e-Literate, August 30\r\n\r\n<span>Ho, A. et al. (2014)\u00a0<\/span><a href=\"http:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=2381263\">HarvardX and MITx: The First Year of Open Online Courses Fall 2012-Summer 2013\u00a0<\/a><span>(HarvardX and MITx Working Paper No. 1), January 21\u00a0<\/span>\r\n\r\n<span>Hollands, F. and Tirthali, D. (2014) <\/span><a href=\"http:\/\/cbcse.org\/wordpress\/wp-content\/uploads\/2014\/05\/MOOCs_Expectations_and_Reality.pdf\">MOOCs: Expectations and Reality<\/a><span> New York: Columbia University Teachers\u2019 College, Center for Benefit-Cost Studies of Education<\/span>\r\n\r\nH\u00fclsmann, T. (2003) Costs without camouflage: a cost analysis of Oldenburg University's \u00a0two graduate certificate programs offered \u00a0as part of the online Master of Distance Education (MDE): a case study, in Bernath, U. and Rubin, E., (eds.) <em>Reflections on Teaching in an Online Program: A Case Study<\/em> Oldenburg, Germany: Bibliothecks-und Informationssystem der Carl von Ossietsky Universit\u00e4t Oldenburg\r\n\r\nJaschik, S. (2013) MOOC Mess, <a href=\"https:\/\/www.insidehighered.com\/news\/2013\/02\/04\/coursera-forced-call-mooc-amid-complaints-about-course\">Inside Higher Education<\/a>, February 4\r\n\r\nKnox, J. (2014) Digital culture clash: 'massive' education in the e-Learning and Digital Cultures\u00a0<em>Distance Education<\/em>, Vol. 35, No. 2\r\n<p class=\"a-size-large a-spacing-none\">Kop, R. (2011) The Challenges to Connectivist Learning on Open Online Networks: Learning Experiences during a Massive Open Online Course\u00a0<em><a href=\"http:\/\/www.irrodl.org\/index.php\/irrodl\/article\/view\/882\/1823\">International\u00a0Review of Research into Open and Distance Learning<\/a><\/em>, Vol. 12, No. 3<\/p>\r\n<p class=\"a-size-large a-spacing-none\">Lave, J. and<span style=\"color: #252525\">\u00a0<\/span>Wenger, E.<span style=\"color: #252525\">\u00a0(1991).\u00a0<\/span><a class=\"external text\" href=\"http:\/\/books.google.com\/?id=CAVIOrW3vYAC\" rel=\"nofollow\" style=\"color: #663366\"><i>Situated Learning: Legitimate Peripheral Participation<\/i><\/a><span style=\"color: #252525\">. Cambridge: Cambridge University Press<\/span><\/p>\r\n<span>Milligan, C., Littlejohn, A. and Margaryan, A. (2013) Patterns of engagement in connectivist MOOCs, <\/span><em><a href=\"http:\/\/jolt.merlot.org\/vol9no2\/milligan_0613.htm\">Merlot Journal of Online Learning and Teaching<\/a><\/em><span>, Vol. 9, No. 2<\/span>\r\n\r\nPiech, C., Huang, J., Chen, Z., Do, C., Ng, A., &amp; Koller, D. (2013)\u00a0<a href=\"http:\/\/www.stanford.edu\/~cpiech\/bio\/papers\/tuningPeerGrading.pdf\"><em>Tuned models of peer assessment in MOOCs<\/em><\/a>. Palo Alto, CA: Stanford University\r\n\r\nRumble, G. (2001) The costs and costing of networked learning, <a href=\"http:\/\/www.c3l.uni-oldenburg.de\/cde\/media\/readings\/rumb01d.pdf\">Journal of Asynchronous Learning Networks<\/a>, Vol. 5, No. 2\r\n<p class=\"a-size-large a-spacing-none\">Suen, H. (2104) <em>Peer assessment for massive open online courses (MOOCs) <a href=\"http:\/\/www.irrodl.org\/index.php\/irrodl\/article\/view\/1680\/2904\">International\u00a0Review of Research into Open and Distance Learning<\/a><\/em>, Vol. 15, No. 3<\/p>\r\n<p class=\"a-size-large a-spacing-none\"><span>University of Ottawa (2013) <\/span><a href=\"http:\/\/www.uottawa.ca\/vr-etudes-academic\/en\/documents\/e-learning-working-group-report.pdf\">Report of the e-Learning Working Group<\/a><span> Ottawa ON: The University of Ottawa<\/span><\/p>\r\n<p class=\"a-size-large a-spacing-none\">van Zundert, M., Sluijsmans, D., van Merri\u00ebnboer, J. (2010). Effective peer assessment processes: Research findings and future directions. <em>Learning and Instruction, 20,<\/em> 270-279<\/p>","rendered":"<blockquote>\n<p style=\"text-align: right\"><em>In-depth analysis by standard academic criteria shows that MOOCs have more academic rigor and are a far more effective teaching methodology than in-house teaching<\/em><\/p>\n<\/blockquote>\n<p style=\"text-align: right\">Benton R. Groves, Ph.D. student<\/p>\n<blockquote>\n<p style=\"text-align: right\"><em>My big concern with xMOOCs is their limitation, as currently designed, for developing the higher order intellectual skills needed in a digital world.<\/em><\/p>\n<\/blockquote>\n<p style=\"text-align: right\">Tony Bates<\/p>\n<h2>5.4.1 The\u00a0research on MOOCs<\/h2>\n<p>Because at the time of writing most MOOCs are less than four\u00a0years old, there are relatively few\u00a0research publications on\u00a0MOOCs, although research activities are now beginning to pick up. Much of the research to date on MOOCs comes from the institutions offering MOOCs, mainly in the form of reports on enrolments, or self-evaluation by instructors. The commercial platform providers such as Coursera and Udacity have provided limited research information\u00a0overall, which is a pity, because they have access to really big data sets. However, MIT and Harvard, the founding partners in edX, are conducting some research, mainly on their\u00a0own courses. There is very little independent research to date on either xMOOCs or cMOOCs.<\/p>\n<p>However, wherever possible, I have tried to use any\u00a0research that has been done that provides insight into the strengths and weaknesses of MOOCs. At the same time, we should be clear that we are discussing a phenomenon that to date has been marked largely by political, emotional and often irrational discourse, and in terms of cumulative hard evidence, we will have to wait for some time.<\/p>\n<p>Lastly, it should be remembered when I am evaluating MOOCs I am applying the criteria of whether MOOCs are likely to lead to the kinds of learning needed in a digital age: in other words, do they help develop the knowledge and skills defined in <a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/chapter\/section-1-3-the-skills-needed-in-a-digital-age\/\">Chapter 1<\/a>?<\/p>\n<h2>5.4.2 Open and free education<\/h2>\n<p>MOOCs, particularly xMOOCs, deliver high quality content from some of the world&#8217;s best universities for free to anyone with a computer and an Internet connection. This in itself is an amazing value proposition. In this sense, MOOCs are an incredibly valuable addition to educational provision.\u00a0Who could argue against this? Certainly not me, so long as the argument for MOOCs goes no further.<\/p>\n<p>However, this is not the only form of open and free education. Libraries, open textbooks and educational broadcasting are also open and free and have been for some time, even if they do not have the same power and reach as Internet-based delivery. There are also lessons we can learn from these earlier\u00a0forms of open and free education that still\u00a0apply to MOOCs.<\/p>\n<p>The first is that these earlier forms of open and free did not replace the need for formal, credit-based education, but were used to supplement or strengthen it. In other words, MOOCs\u00a0are a tool for continuing and informal education, which has high value in its own right. As we shall see though they work best when people are already reasonably well educated.<\/p>\n<p>The problem comes when it is argued that because MOOCs are open and free to end-users, they will inevitably force down the cost of conventional higher education, or eliminate the need for it altogether, especially in developing\u00a0countries (see the Friedman comment at the beginning of this chapter.)<\/p>\n<p>There have been many attempts in the past to use educational broadcasting and satellite broadcasting in developing\u00a0countries (see Bates, 1985), and they all failed substantially\u00a0to increase access or reduce cost\u00a0for a variety of reasons, the most important being:<\/p>\n<ul>\n<li>the high cost of ground equipment (including security from theft or damage);<\/li>\n<li>the need for local support for learners without high levels of education, and the\u00a0high cost of local, &#8216;ground&#8217; support;<\/li>\n<li>the need to adapt to the culture of the receiving countries;<\/li>\n<li>the difficulty of covering the operational costs of management and administration, especially for assessment, qualifications and local accreditation.<\/li>\n<\/ul>\n<p>Also the priority in most developing\u00a0countries is not for courses from high-level Stanford University professors, but for\u00a0programs for high schools. Finally,\u00a0although<span>\u00a0mobile phones are widespread in Africa, they operate on very narrow bandwidths. For instance, it costs US$2 to download a typical YouTube video \u2013 equivalent to a day\u2019s salary for many Africans. Streamed video lectures then have limited applicability.<\/span><\/p>\n<p>This is not to say that MOOCs could not be valuable in developing\u00a0countries, but\u00a0this will mean:<\/p>\n<ul>\n<li>being realistic as to what they\u00a0can actually deliver;<\/li>\n<li>working in partnership with educational institutions and systems and other partners in developing countries;<\/li>\n<li>ensuring that the necessary local\u00a0support &#8211; which costs real money &#8211; is put in place;<\/li>\n<li>adapting the design, content and delivery of MOOCs to the cultural and economic requirements of\u00a0those countries.<\/li>\n<\/ul>\n<p>Furthermore, MOOCs are not always open as in the sense of open educational resources. Coursera and Udacity for instance offer limited access to their material for re-use without permission. On other more open platforms, such as edX, individual faculty or institutions may restrict re-use of material. Lastly, many MOOCs exist for only one or two years then disappear, which limits their use as open educational resources for re-use in other courses or programs.<\/p>\n<p>Finally, although MOOCs are in the main free for participants, they are not without substantial cost to MOOC providers, an issue that will be discussed in more detail in Section 5.4.8.<\/p>\n<h2>5.4.3 The audience that MOOCs mainly serve<\/h2>\n<p>In <a href=\"http:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=2381263\">a\u00a0research report<\/a> from Ho et al. (2014), researchers at Harvard University and MIT found that on the first 17 MOOCs offered through edX,\u00a0<span>66\u00a0per cent\u00a0of all participants, and 74\u00a0per cent\u00a0of all who obtained a certificate, have a bachelor\u2019s degree or above, 71 per cent were male, and the average age was 26. This and other studies also found that a high proportion of\u00a0participants\u00a0came from outside the USA, ranging from 40-60\u00a0per cent\u00a0of all participants, indicating strong interest\u00a0internationally in\u00a0open access to high quality university teaching.<\/span><\/p>\n<p>In a study\u00a0based on over 80 interviews in 62 institutions &#8216;active in the MOOC space&#8217;, <a href=\"http:\/\/cbcse.org\/wordpress\/wp-content\/uploads\/2014\/05\/MOOCs_Expectations_and_Reality.pdf\">Hollands and Tirthali (2014)<\/a>, researchers at Columbia University Teachers&#8217; College, found that:<\/p>\n<blockquote><p><em>Data from MOOC platforms indicate that MOOCs are providing educational opportunities to millions of individuals across the world. However, most MOOC participants are already well-educated and employed, and only a small fraction of them fully engages with the courses. Overall, the evidence suggests that MOOCs are currently falling far short of \u201cdemocratizing\u201d education and may, for now, be doing more to increase gaps in access to education than to diminish them.<\/em><\/p><\/blockquote>\n<p>Thus MOOCs, as is common with most forms of university continuing education, cater to the better educated, older and employed sectors of society.<\/p>\n<h2>5.4.4 Persistence and commitment<\/h2>\n<p>The edX researchers (Ho et al., 2014)\u00a0identified different levels of commitment as follows across 17 edX MOOCs:<\/p>\n<ul>\n<li><strong>only registered<\/strong>: registrants who never access the courseware (35\u00a0per cent);<\/li>\n<li><strong>only viewed<\/strong>: non-certified registrants who access the courseware, accessing less than half of the available chapters (56\u00a0per cent);<\/li>\n<li><strong>only explored<\/strong>: non-certified registrants who access more than half of the available chapters in the courseware, but did not get a certificate (4\u00a0per cent);<\/li>\n<li><strong>certified<\/strong>: registrants who earn a certificate in the course (5\u00a0per cent).<\/li>\n<\/ul>\n<p>Hill (2013) has identified five types of participants in Coursera courses:<\/p>\n<p><a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg\" alt=\"Print\" width=\"755\" height=\"26\" class=\"aligncenter size-full wp-image-1668\" srcset=\"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg 755w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line-300x10.jpg 300w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line-65x2.jpg 65w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line-225x8.jpg 225w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line-350x12.jpg 350w\" sizes=\"auto, (max-width: 755px) 100vw, 755px\" \/><\/a><\/p>\n<figure id=\"attachment_784\" aria-describedby=\"caption-attachment-784\" style=\"width: 746px\" class=\"wp-caption aligncenter\"><a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/Hills-MOOC-participants-2.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/Hills-MOOC-participants-2-1024x750.jpg\" alt=\"\u00a9 Phil Hill, 2013\" width=\"746\" height=\"548\" class=\"wp-image-784\" srcset=\"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/Hills-MOOC-participants-2-300x219.jpg 300w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/Hills-MOOC-participants-2-65x47.jpg 65w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/Hills-MOOC-participants-2-225x164.jpg 225w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/Hills-MOOC-participants-2-350x256.jpg 350w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/Hills-MOOC-participants-2.jpg 1263w\" sizes=\"auto, (max-width: 746px) 100vw, 746px\" \/><\/a><figcaption id=\"caption-attachment-784\" class=\"wp-caption-text\">Figure 5.4.4.1 \u00a9 Phil Hill, 2013<\/figcaption><\/figure>\n<p><a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg\" alt=\"Print\" width=\"755\" height=\"13\" class=\"aligncenter size-full wp-image-1580\" srcset=\"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg 755w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2-300x5.jpg 300w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2-65x1.jpg 65w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2-225x4.jpg 225w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2-350x6.jpg 350w\" sizes=\"auto, (max-width: 755px) 100vw, 755px\" \/><\/a><\/p>\n<p>Engle\u00a0(2014) found\u00a0similar patterns for the University of British Columbia MOOCs on Coursera (also replicated in other studies):<\/p>\n<ul>\n<li>of those that initially sign up, between one third and a half do not participate in any other active way;<\/li>\n<li>of those that participate in at least one activity, between 5-10\u00a0per cent\u00a0go on to successfully complete a certificate.<\/li>\n<\/ul>\n<p>Those going on to achieve certificates usually are within the 5-10\u00a0per cent\u00a0range of those that sign up and in the 10-20\u00a0per cent\u00a0range for those who actively engaged with the MOOC at least once. Nevertheless, the numbers obtaining certificates are still large in absolute terms: over 43,000 across 17 courses on\u00a0edX and 8,000 across four courses at UBC (between 2,000-2,500 certificates per course).<\/p>\n<p><a href=\"http:\/\/jolt.merlot.org\/vol9no2\/milligan_0613.htm\">Milligan et al<\/a>. (2013) found a similar pattern of commitment in cMOOCs, from interviewing a small sample of participants (29 out of 2,300 registrants) about halfway through a\u00a0cMOOC:<\/p>\n<ul>\n<li>passive participants: in Milligan&#8217;s study these were those that felt lost in the MOOC and rarely but occasionally logged in;<\/li>\n<li>lurkers: they were actively following the course but did not engage in any of the activities (just under half those interviewed);<\/li>\n<li>active participants (again, just under half those interviewed) who were fully engaged in the course activities.<\/li>\n<\/ul>\n<p><span>MOOCs need to be judged for what they are, a somewhat unique \u2013 and valuable \u2013 form of non-formal education.\u00a0<\/span>These results are very similar to research into non-formal educational broadcasts (e.g. the History Channel). One would not expect a viewer to watch every episode of a History Channel series then take an exam at the end. Ho et al. (p.13) produced the following diagram to show the different levels of commitment to xMOOCs:<\/p>\n<p><a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg\" alt=\"Print\" width=\"465\" height=\"16\" class=\"aligncenter wp-image-1668\" srcset=\"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg 755w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line-300x10.jpg 300w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line-65x2.jpg 65w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line-225x8.jpg 225w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line-350x12.jpg 350w\" sizes=\"auto, (max-width: 465px) 100vw, 465px\" \/><\/a><\/p>\n<figure id=\"attachment_780\" aria-describedby=\"caption-attachment-780\" style=\"width: 457px\" class=\"wp-caption aligncenter\"><a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/edX-participants-2.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/edX-participants-2.jpg\" alt=\"Ho et al., 2014\" width=\"457\" height=\"587\" class=\"wp-image-780\" srcset=\"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/edX-participants-2.jpg 406w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/edX-participants-2-234x300.jpg 234w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/edX-participants-2-65x83.jpg 65w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/10\/edX-participants-2-225x288.jpg 225w\" sizes=\"auto, (max-width: 457px) 100vw, 457px\" \/><\/a><figcaption id=\"caption-attachment-780\" class=\"wp-caption-text\">Figure 5.4.4.2 Level of participation in MOOCs\u00a0 \u00a9\u00a0Ho et al., 2014<\/figcaption><\/figure>\n<p><a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg\" alt=\"Print\" width=\"465\" height=\"8\" class=\"aligncenter wp-image-1580\" srcset=\"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg 755w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2-300x5.jpg 300w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2-65x1.jpg 65w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2-225x4.jpg 225w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2-350x6.jpg 350w\" sizes=\"auto, (max-width: 465px) 100vw, 465px\" \/><\/a><\/p>\n<p>Now compare that to what I wrote in 1985 about educational broadcasting in Britain (Bates, 1985):<\/p>\n<blockquote><p><span>(p.99): <em>At the centre of the onion is a small core of fully committed students who work through the whole course, and, where available, take an end-of-course assessment or examination. Around the small core will be a rather larger layer of students who do not take any examination but do enrol with a local class or correspondence school. There may be an even larger layer of\u00a0students who, as well as watching and listening, also buy the accompanying textbook, but who do not enrol in any courses. Then, by far the largest group, are those that just watch or listen to the programmes. Even within this last group, there will be considerable variations, from those who watch or listen fairly regularly, to those, again a much larger number, who watch or listen to just one programme.\u00a0<\/em><\/span><\/p><\/blockquote>\n<p>I also wrote (p.100):<\/p>\n<blockquote><p><em>A sceptic may say that the only ones who can be said to have learned effectively are the tiny minority that worked right through the course and successfully took the final assessment\u2026A counter argument would be that broadcasting can be considered successful if it merely attracts viewers or listeners who might otherwise have shown no interest in the topic; it is the numbers exposed to the material that matter\u2026the key issue then is whether broadcasting does attract to education those who would not otherwise have been interested, or merely provides yet another opportunity for those who are already well educated\u2026There is a good deal of evidence that it is still the better educated in Britain and Europe that make the most use of non-formal educational broadcasting.<\/em><\/p><\/blockquote>\n<p><span>Exactly the same could be said about MOOCs. In a digital age where easy and open access to new knowledge is critical for those working in knowledge-based industries, MOOCs will be one valuable source or means of accessing that knowledge. The issue is though whether there are more effective ways to do this.\u00a0<\/span>Thus MOOCs can be considered\u00a0a useful &#8211; but not really revolutionary &#8211;\u00a0contribution to non-formal continuing education.<\/p>\n<h2>5.4.5 What do students learn in MOOCs?<\/h2>\n<p>This is a much more difficult question to answer, because so little of the research to date (2014) has tried to answer this question. (One reason, as we shall see in the next section, is that assessment of learning in MOOCs remains a major challenge). There are at least two kinds of study: quantitative\u00a0studies that seek to quantify learning gains; and qualitative studies that describe the experience of learners within MOOCs, which indirectly provide some insight into what they have learned.<\/p>\n<p>At the time of writing, the most quantitative\u00a0study of learning in MOOCs has been\u00a0by <a href=\"http:\/\/www.irrodl.org\/index.php\/irrodl\/article\/view\/1902\/3009\">Colvin et al. (2014)<\/a>, who investigated &#8216;conceptual learning&#8217; in an MIT Introductory Physics MOOC. They compared learner performance not only between different sub-categories of learners within the MOOC, such as those with no physics or math background with those such as physic teachers who had considerable prior knowledge, but also with on-campus students taking the same\u00a0curriculum in\u00a0a traditional campus teaching format. In essence, the study found no significant differences in learning gains between or within the two types of teaching, but it should be noted that the on-campus students were students who had failed an earlier version of the course and were retaking it.<\/p>\n<p>This research is a classic example of the no significant difference in comparative studies in educational technology; other variables, such as differences in the types of students, were as\u00a0important as\u00a0the mode of delivery.\u00a0Also<span>, this MOOC design represents\u00a0a behaviourist-cognitivist approach to learning that places heavy emphasis on correct answers to conceptual questions. It doesn&#8217;t attempt to\u00a0develop the skills needed in a digital age as identified in Chapter 1.<\/span><\/p>\n<p>There have been far more studies of the experience of learners within MOOCs, particularly focusing on the discussions within MOOCs (see for instance, Kop, 2011). In general (although there are exceptions), discussions are unmonitored, and it is left to participants to make connections and respond to other students comments. However<span>, there are some strong criticisms\u00a0of the effectiveness of the discussion element of MOOCs for\u00a0developing the high-level conceptual analysis\u00a0required for academic learning. To develop deep, conceptual learning, there is a need in most cases for intervention by a subject expert to clarify misunderstandings or misconceptions, to provide accurate feedback, \u00a0to ensure that the criteria for academic learning, such as use of evidence, clarity of argument, and so on, are being met, and to ensure\u00a0the necessary input and guidance to seek deeper understanding (see Harasim, 2013).<\/span><\/p>\n<p><span>Furthermore, the more massive the course, the more likely participants\u00a0are to feel \u2018overload, anxiety and a sense of loss\u2019, if there is not some instructor intervention or structure imposed (Knox, 2014). Firmin et\u00a0al. (2014) have shown that when there is some form of instructor \u2018encouragement and support of student effort and engagement\u2019, results improve for all participants in MOOCs. Without a structured role for subject experts,\u00a0participants are faced with a wide variety of quality in terms of comments and feedback from other participants.\u00a0There is again a great deal of research on the conditions necessary for the successful conduct of collaborative and co-operative group learning (see for instance, Dillenbourg, 1999, Lave and Wenger, 1991), and\u00a0these findings certainly have not been generally applied to the management of MOOC discussions to date.\u00a0<\/span><\/p>\n<p><span>One\u00a0counter argument is that at least cMOOCs develop a new form of learning based on networking and collaboration that is essentially different from academic learning, and MOOCs are thus\u00a0more appropriate to the needs of learners in a digital age. Adult participants in particular, it is claimed by Downes and Siemens, have the ability to self-manage the development of high level conceptual learning. \u00a0MOOCs are \u2018demand\u2019 driven, meeting the interests of individual students who seek out others with similar interests and the necessary expertise to support them in their learning, and for many this interest may well not include the need for deep, conceptual learning but more likely the appropriate applications of prior\u00a0knowledge in new or specific contexts.\u00a0<\/span>MOOCs do appear to work best for those who already have a high level of education and therefore bring many of the conceptual skills developed in formal education with them when they join a MOOC, and therefore contribute to helping those who come without such prior knowledge or skills.<\/p>\n<p><span>Over time, as more experience is gained, MOOCs are likely to incorporate\u00a0and\u00a0adapt some of the findings from research on smaller group work to the\u00a0much larger numbers in MOOCs. For instance, some MOOCs are using &#8216;volunteer&#8217; or community tutors (Dillenbourg, 2014). The US State Department has organized MOOC camps\u00a0through\u00a0<span>US missions and consulates abroad\u00a0<\/span>to mentor MOOC participants. The camps include Fulbright scholars and embassy staff who lead discussions on content and topics for MOOC participants in countries abroad (Haynie, 2014). Some MOOC providers, such as the University of British Columbia, pay a small cohort of academic\u00a0assistants to monitor and contribute to the MOOC discussion forums (Engle, 2014).\u00a0Engle reported that the<span>\u00a0use of academic assistants, as well as limited but effective interventions from the instructors themselves, made the UBC MOOCs more interactive and engaging. However, paying for people to monitor and support MOOCs will of course increase the cost to providers. Consequently, MOOCs are likely to develop\u00a0new automated ways to manage\u00a0discussion effectively in very large groups. The University of Edinburgh is experimenting with automated &#8216;teacherbots&#8217; that crawl through online discussion forums and direct predetermined comments to students identified as needing help or encouragement\u00a0(Bayne, 2014).\u00a0<\/span><\/span><\/p>\n<p><span><span>These results and approaches are\u00a0consistent\u00a0with\u00a0prior research on the importance of instructor presence for successful for-credit online learning.\u00a0<\/span>In the meantime, though, there is much work still to be done if MOOCs are to provide the support and structure needed to ensure deep, conceptual learning where this does not already exist in students. The development of the skills needed in a digital age is likely to be an even greater challenge when dealing with massive numbers. However, we need much more research into what participants actually learn in MOOCs and under what conditions before any firm conclusions can be drawn.<\/span><\/p>\n<h2>5.4.6 Assessment<\/h2>\n<p>Assessment of the massive numbers of participants in MOOCs has proved to be a major challenge. It is a complex topic that can be dealt with only briefly here. However, <a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/chapter\/5-8-assessment-of-learning\/\">Appendix 1, Section 8<\/a>\u00a0provides a\u00a0general analysis\u00a0of different types of assessment, and <a href=\"http:\/\/www.irrodl.org\/index.php\/irrodl\/article\/view\/1680\/2904\">Suen (2014)<\/a> provides a comprehensive and balanced overview of the way assessment has been used in MOOCs to date. This section draws heavily on Suen&#8217;s paper.<\/p>\n<h3 style=\"text-align: left\">5.4.6.1 Computer marked assignments<\/h3>\n<p>Assessment to date in MOOCs has been primarily of two kinds. The first is based on quantitative multiple-choice tests, or response boxes where formulae or &#8216;correct code&#8217; can be entered and automatically checked. Usually participants are given immediate automated feedback on their answers, ranging from simple right or wrong answers to more complex responses depending on the type of response checked, but in all cases, the process is usually fully automated.<\/p>\n<p>For straight testing of facts, principles, formulae, equations and other forms of conceptual learning where there are clear, correct answers, this works well. In fact, multiple choice computer marked assignments were used by the UK Open University as long ago as the 1970s, although the means to give immediate online feedback were not available then. However, this method of assessment\u00a0is limited for testing deep or &#8216;transformative&#8217; learning,\u00a0and particularly weak for assessing\u00a0the intellectual skills needed in a digital age, such as creative or original thinking.<\/p>\n<h3 style=\"text-align: left\">5.4.6.2 Peer\u00a0assessment<\/h3>\n<p>The second type of assessment that has been tried in MOOCs has been peer assessment, where participants assess each other&#8217;s work. Peer assessment is not new. It has been successfully used for formative assessment in traditional classrooms and in some online teaching for credit (Falchikov and Goldfinch, 2000;\u00a0van Zundert et al., 2010). More importantly, peer assessment is seen as a powerful way to improve deep understanding and knowledge through the process of students evaluating the work of others, and at the same time, it can be useful for developing some of the skills needed in a digital age, such as critical thinking, for those participants assessing other participants.<\/p>\n<p>However, a key feature of the successful use of peer assessment has been the close involvement of an instructor or teacher, in providing benchmarks, rubrics or criteria for assessment, and for monitoring and adjusting peer assessments to ensure consistency and a match with the benchmarks\u00a0set by the instructor. Although\u00a0an instructor can provide the benchmarks and rubrics in MOOCs,\u00a0close monitoring of the multiple peer assessments is difficult if not impossible with the very large numbers of participants. As a result, MOOC participants\u00a0often become incensed at being randomly assessed by other participants who\u00a0may not and often do not\u00a0have\u00a0the knowledge or ability to give a &#8216;fair&#8217; or accurate assessment of a participant&#8217;s work.<\/p>\n<p>Various attempts to get round the limitations of peer assessment in MOOCs have been tried such as calibrated peer reviews, based on averaging all the peer ratings, and Bayesian post hoc stabilization (Piech at al. 2013), but although these statistical techniques reduce the error (or spread) of peer review somewhat they still do not remove the problems of systematic errors of judgement in raters due to misconceptions. This is particularly a problem where a majority of participants fail to understand key concepts in a\u00a0MOOC, in which case peer assessment becomes the blind leading the blind.<\/p>\n<h3 style=\"text-align: left\"><strong>5.4.6.3 Automated essay scoring<\/strong><\/h3>\n<p>This is another area where there have been attempts to automate scoring (Balfour, 2013). Although such methods are increasingly sophisticated they are currently limited in terms of accurate assessment to measuring primarily technical writing skills, such as grammar, spelling and sentence construction. Once again they do not measure accurately essays where higher level intellectual skills are demonstrated.<\/p>\n<h3 style=\"text-align: left\">5.4.6.4 Badges and certificates<\/h3>\n<p>Particularly in xMOOCs, participants may be awarded a certificate or a &#8216;badge&#8217; for successful completion of the MOOC, based on a final test (usually computer-marked) which measures the level of learning in a course.<\/p>\n<p>T<span>he American Council on Education (ACE), which\u00a0<span>represents the presidents of U.S. accredited, degree-granting institutions,<\/span>\u00a0recommended offering credit for five courses on the\u00a0Coursera MOOC platform.\u00a0<\/span>However, according to the person responsible for the review process (Book, 2013):<\/p>\n<blockquote><p><em>what the ACE accreditation does is merely accredit courses from institutions that are already accredited.\u00a0The review process doesn\u2019t evaluate learning outcomes, but is a course content focused review thus obviating all the questions about effectiveness of the pedagogy in terms of learning outcomes.\u00a0<\/em><\/p><\/blockquote>\n<p>Indeed, most of the institutions offering MOOCs will not accept their own certificates for admission or credit within their own, campus-based programs. Probably nothing says more about the confidence in the quality of the assessment than this failure of MOOC providers to recognize their own teaching.<\/p>\n<h3 style=\"text-align: left\">5.4.6.5 The intent behind assessment<\/h3>\n<p>To evaluate assessment in MOOCs requires an examination of the intent behind assessment. There are many different purposes behind assessment (see <a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/chapter\/5-8-assessment-of-learning\/\">Appendix 1, Section 8<\/a>). Peer assessment and immediate feedback on computer-marked tests can be extremely valuable for formative assessment, enabling participants to see what they have understood and to help develop further their understanding of key concepts. In cMOOCs, as Suen points out, learning is measured as the communication that takes place between MOOC\u00a0participants, resulting in crowdsourced validation of knowledge &#8211; it&#8217;s what the sum of all the participants come to believe to be true as a result of participating in the MOOC, so formal assessment is unnecessary. However, what is learned in this way\u00a0is not necessarily <em>academically<\/em> validated knowledge, which to be fair, is not the concern of cMOOC\u00a0proponents.<\/p>\n<p>Academic assessment is a form of currency, related not only to measuring student achievement but also affecting student mobility (for example, entrance to graduate school) and perhaps more importantly employment opportunities and promotion. From a learner&#8217;s perspective, the validity of the currency &#8211; the recognition and transferability of the qualification &#8211; is essential. To date, MOOCs have been unable to demonstrate that they are able to assess accurately the learning achievements of participants beyond comprehension and knowledge of ideas, principles and processes (recognizing that there is some value in this alone). What MOOCs\u00a0have not been able to demonstrate is that they can either develop or assess deep understanding or the intellectual skills required in a digital age. Indeed, this\u00a0may not be possible within the constraints of massiveness, which is their major distinguishing feature from other forms of online learning.<\/p>\n<h2>5.4.7 Branding<\/h2>\n<p>Hollands and Tirthali (2014) in their survey on institutional expectations for MOOCs, found that building and maintaining brand was the second most important reason for institutions launching MOOCs (the most important was extending reach, which can also be seen as partly a branding exercise). Institutional branding through the use of MOOCs has\u00a0been helped by elite Ivy League universities such as Stanford, MIT and Harvard leading the charge, and by Coursera limiting access to its platform to only &#8216;top tier&#8217; universities. This of course has led to a bandwagon effect, especially since many of the universities launching\u00a0MOOCs had previously disdained to move into credit-based online learning. MOOCs provided a way for these elite institutions to jump to the head of the queue in terms of status as\u00a0&#8216;innovators&#8217; of online learning, even though they arrived late to the party.<\/p>\n<p>It obviously makes sense for institutions to use MOOCs to bring their areas of specialist expertise to a much wider public, such as the University of Alberta offering a MOOC on dinosaurs, MIT on electronics, and Harvard on Ancient Greek Heroes. MOOCs\u00a0certainly help to widen knowledge of\u00a0the quality of an individual professor (who is\u00a0usually delighted to reach more students in one MOOC than in a lifetime of on-campus teaching). MOOCs are also a good way to give a glimpse of the quality\u00a0of courses and programs offered by an institution.<\/p>\n<p>However, it is difficult to measure the real impact of MOOCs on branding. As Hollands and Tirthali put it:<\/p>\n<blockquote><p><em>While many institutions have received significant media attention as a result of their MOOC activities, isolating and measuring impact of any new initiative on brand is a difficult exercise. Most institutions are only just beginning to think about how to capture and quantify branding-related benefits.<\/em><\/p><\/blockquote>\n<p>In particular,\u00a0these elite institutions do not need MOOCs to boost the number of applicants for their campus-based programs\u00a0(none to date is willing to accept successful completion of a MOOC for admission to credit programs), since\u00a0elite institutions have no difficulty in attracting already highly qualified students.<\/p>\n<p>Furthermore, once every other institution starts offering MOOCs, the branding effect gets lost to some extent. Indeed, exposing poor quality teaching or course planning to many thousands can have a negative impact on an institution&#8217;s brand, as\u00a0Georgia Institute of Technology found when one of its MOOCs crashed and burned<a href=\"https:\/\/www.insidehighered.com\/news\/2013\/02\/04\/coursera-forced-call-mooc-amid-complaints-about-course\"> (Jaschik, 2013).<\/a>\u00a0However, by and large, most MOOCs succeed in the sense of bringing an institution&#8217;s reputation in terms of knowledge and expertise\u00a0to many more people than\u00a0it would through\u00a0any other form of teaching or publicity.<\/p>\n<h2>5.4.8 Costs and economies of scale<\/h2>\n<p><a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg\" alt=\"Print\" width=\"755\" height=\"26\" class=\"aligncenter size-full wp-image-1668\" srcset=\"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line.jpg 755w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line-300x10.jpg 300w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line-65x2.jpg 65w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line-225x8.jpg 225w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-large-line-350x12.jpg 350w\" sizes=\"auto, (max-width: 755px) 100vw, 755px\" \/><\/a><\/p>\n<figure id=\"attachment_884\" aria-describedby=\"caption-attachment-884\" style=\"width: 740px\" class=\"wp-caption aligncenter\"><a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/11\/Fixed-and-variable-costs-2.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/11\/Fixed-and-variable-costs-2.jpg\" alt=\"The MOOC value proposition is that MOOCs can eliminate the variable costs of course delivery. Image: \u00a9 OpenTuition.com, 2014\" width=\"740\" height=\"393\" class=\"wp-image-884\" srcset=\"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/11\/Fixed-and-variable-costs-2.jpg 607w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/11\/Fixed-and-variable-costs-2-300x159.jpg 300w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/11\/Fixed-and-variable-costs-2-65x34.jpg 65w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/11\/Fixed-and-variable-costs-2-225x119.jpg 225w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2014\/11\/Fixed-and-variable-costs-2-350x185.jpg 350w\" sizes=\"auto, (max-width: 740px) 100vw, 740px\" \/><\/a><figcaption id=\"caption-attachment-884\" class=\"wp-caption-text\">Figure 5.4.8 The MOOC value proposition is that MOOCs can eliminate the variable costs of course delivery.\u00a0Image: \u00a9 OpenTuition.com, 2014<\/figcaption><\/figure>\n<p><a href=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg\" alt=\"Print\" width=\"755\" height=\"13\" class=\"aligncenter size-full wp-image-1580\" srcset=\"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2.jpg 755w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2-300x5.jpg 300w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2-65x1.jpg 65w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2-225x4.jpg 225w, https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-content\/uploads\/sites\/29\/2015\/01\/Wei-2-350x6.jpg 350w\" sizes=\"auto, (max-width: 755px) 100vw, 755px\" \/><\/a><\/p>\n<p>One main strength claimed for\u00a0MOOCs is that they are free to participants. Once again we shall see this is more true in principle than in practice, because\u00a0MOOC\u00a0providers may charge a range of fees, especially for assessment. Furthermore, although MOOCs may be free for participants, they are\u00a0not\u00a0without substantial cost to the provider institutions. Also, there are large differences in the costs of xMOOCs and cMOOCs, the latter being generally much cheaper to develop, although there are still some opportunity or actual costs even for cMOOCs.<\/p>\n<p>Once again, there is very little information to date on the actual costs of designing and delivering a MOOC\u00a0as\u00a0\u00a0there are not\u00a0enough cases at the moment to draw firm conclusions about the costs of MOOCs. However\u00a0we do have some data. The <a href=\"http:\/\/www.uottawa.ca\/vr-etudes-academic\/en\/documents\/e-learning-working-group-report.pdf\">University of Ottawa<\/a>\u00a0(2013) estimated the cost of developing an xMOOC, based on figures provided to the university by Coursera, and on their own knowledge of the cost of developing online courses for credit, at around $100,000.<\/p>\n<p><a href=\"http:\/\/flexible.learning.ubc.ca\/files\/2014\/09\/MOOC-Report.pdf\">Engle (2014)<\/a> has reported on the actual cost of five MOOCs from the University of British Columbia. (In essence, there were really four UBC MOOCs, as one was in two shorter parts.) There are two important features concerning the UBC MOOCs that do not necessarily apply to other MOOCs. First, the UBC MOOCs used a wide variety of video production methods, from full studio production to desktop recording, so development costs varied considerably, depending on the sophistication of the video production technique. Second, the UBC MOOCs made extensive use of paid academic assistants, who monitored discussions and adapted or changed\u00a0course materials as a result of student feedback, so there were substantial delivery costs as well.<\/p>\n<p><span>Appendix B of the UBC report gives a pilot total of $217,657, but this excludes academic assistance or, perhaps the most significant cost, instructor time. Academic assistance came to 25\u00a0per cent\u00a0of the overall cost in the first year (<em>excluding<\/em> the cost of faculty). Working from the video\u00a0production costs ($95,350) and the proportion of costs (44\u00a0per cent) devoted to video\u00a0production in Figure 1 in the report, I estimate the direct cost at $216,700, or approximately $54,000\u00a0per MOOC,\u00a0<em>excluding<\/em> faculty time and co-ordination support (that is,\u00a0excluding program administration and overheads), but including academic assistance. However, the range of cost is almost as important. The video production costs for the MOOC which\u00a0used intensive studio production were more than six times\u00a0the video production costs of one of the other MOOCs.<\/span><\/p>\n<p>The main cost factors or variables in <em>credit-based<\/em> online and distance learning are relatively well understood, from previous research by Rumble (2001) and H\u00fclsmann\u00a0(2003). Using similar\u00a0costing methodology, I tracked and analysed the cost of an online master&#8217;s program at the University of British Columbia over a seven year period (Bates and Sangr\u00e0, 2011). This program used mainly a learning management system as the core technology, with instructors both developing the course and providing online learner support and assessment, assisted where necessary by extra adjunct faculty for handling larger class enrolments.<\/p>\n<p>I found in my analysis of the costs of the UBC program that in 2003, development costs were approximately $20,000 to $25,000 per course. However, over a seven year period, course development\u00a0constituted less than 15\u00a0per cent\u00a0of the total cost, and occurred mainly in the first year or so of the program. Delivery costs, which included providing online learner support and student assessment, constituted more than a third of the total cost, and of course continued each year the course was offered. Thus in credit-based online learning, delivery costs tend to be more than double the development costs over the life of a program.<\/p>\n<p>The main difference then between MOOCs, credit-based online teaching,\u00a0and campus-based teaching is that in principle\u00a0MOOCs eliminate all delivery\u00a0costs, because MOOCs do not provide learner support or instructor-delivered assessment, although again in practice this is not always true.<\/p>\n<p>There is also clearly a large opportunity cost involved in offering xMOOCs. By definition, the most highly valued faculty are involved in offering MOOCs. In a large research university, such faculty are likely to have, at a maximum, a teaching load of four to six courses a year. Although most instructors volunteer to do MOOCs, their time is limited. Either it means dropping one credit course for at least one semester, equivalent to 25 per cent or more of their teaching load, or xMOOC development and delivery\u00a0replaces time spent doing research. Furthermore, unlike credit-based courses, which run from anywhere between five to seven years, MOOCs are often offered only once or twice.<\/p>\n<p>However one looks at it, the cost of xMOOC development, <em>without<\/em> including the time of the\u00a0MOOC instructor, tends to be almost double the cost of developing an online credit course using a learning management system, because of the use of video in MOOCs.\u00a0If the cost of the instructor is included, xMOOC production costs come closer to three times that of a similar length online credit course, especially given\u00a0the extra time faculty tend put in for such a public demonstration of their teaching in a MOOC. xMOOCs could (and some do)\u00a0use cheaper production methods, such as an LMS instead of video, for content delivery, or using and re-editing video recordings of classroom lectures via lecture capture.<\/p>\n<p>Without learner support or academic\u00a0assistance, though, delivery costs for MOOCs are zero, and this is where the huge potential for savings exist. If the cost per participant is calculated the unit costs are very low. Even if the cost per student successfully obtaining an end of course certificate is calculated it will be many times lower than the cost of an online or campus-based successful student. If we take a MOOC costing roughly $100,000 to develop, and 5,000 participants\u00a0complete the end of course certificate, the average cost per\u00a0successful participant is $20. However, this assumes that the same type\u00a0of knowledge and skills is being assessed for both a MOOC and for a graduate masters program; usually\u00a0this not the case.<\/p>\n<p>The\u00a0issue then is whether MOOCs can succeed without the cost of learner support and human assessment, or more likely, whether MOOCs can substantially reduce delivery costs through automation without loss of quality in learner performance. There is no evidence to date though that they can do this in terms of higher order learning skills and &#8216;deep&#8217; knowledge. To assess this kind of learning requires setting assignments that test such knowledge, and such assessments\u00a0usually need human marking, which then adds to cost. We also\u00a0know from prior research from\u00a0successful online credit programs that active instructor online presence is a critical factor for successful online learning. Thus adequate learner support and assessment\u00a0remains a major challenge for MOOCs. MOOCs then are a good way to teach\u00a0certain levels of knowledge but will have major structural problems in teaching other types of knowledge. Unfortunately, it is the type of knowledge most needed in a digital world that MOOCs struggle to teach.<\/p>\n<p>In terms of sustainable business models, the elite universities have been able to move into xMOOCs because of generous donations from private foundations and use of endowment funds, but these forms of funding are limited for most institutions. Coursera and Udacity have the opportunity to develop successful business models through various means, such as charging MOOC provider\u00a0institutions\u00a0for use of their platform, by collecting fees for badges or certificates, through the sale of participant data, through corporate sponsorship, or through direct advertising.<\/p>\n<p>However, particularly for publicly funded universities or colleges, most of these sources of income are not available or permitted, so it is hard to see how they can begin to recover the cost of a substantial\u00a0investment in MOOCs, even with &#8216;cannibalising&#8217; MOOC material for on-campus use. Every time a MOOC is offered, this takes away resources that could be used for online credit programs. Thus institutions are faced with some hard decisions about where to invest their resources for online learning. The case for putting scarce resources\u00a0into MOOCs is far from clear, unless some way can be found to give credit for successful MOOC completion.<\/p>\n<h2>5.4.9\u00a0Summary of strengths and weaknesses<\/h2>\n<p>The main points of this analysis of the strengths and weaknesses of MOOCs can be summarised as follows:<\/p>\n<h3 style=\"text-align: left\">5.4.9.1 Strengths<\/h3>\n<ul>\n<li><span>MOOCs, particularly xMOOCs, deliver high quality content from some of the world&#8217;s best universities for free to anyone with a computer and an Internet connection;<\/span><\/li>\n<li>MOOCs can be useful for opening access to high quality content, particularly in developing\u00a0countries, but to do so successfully will require a good deal of adaptation, and substantial investment in local support and partnerships;<\/li>\n<li>MOOCs are valuable for developing basic conceptual learning, and for creating large online communities of interest or practice;<\/li>\n<li>MOOCs are an extremely valuable form of lifelong learning and\u00a0continuing education;<\/li>\n<li>MOOCs have forced conventional and especially elite institutions to reappraise their strategies towards online and open learning;<\/li>\n<li>institutions have been able to extend their brand and status by making public their expertise and excellence in certain academic areas;<\/li>\n<li>MOOCs main value proposition is\u00a0to eliminate through computer automation and\/or peer-to-peer communication\u00a0the very large variable costs in higher education associated with providing learner support and quality assessment.<\/li>\n<\/ul>\n<h3 style=\"text-align: left\">5.4.9.2 Weaknesses<\/h3>\n<ul>\n<li>the high registration numbers for MOOCs are misleading; less than half of registrants actively participate, and of these, only a small proportion successfully complete the course; nevertheless, absolute numbers are still higher than for conventional courses;<\/li>\n<li>MOOCs are expensive to develop, and although commercial organisations offering MOOC platforms have opportunities for sustainable business models, it is difficult to see how publicly funded higher education institutions can develop sustainable business models for MOOCs;<\/li>\n<li>MOOCs tend to attract\u00a0those with already a high level of education, rather than widen access;<\/li>\n<li>MOOCs so far have been limited in the ability to develop high level academic learning, or the high level intellectual skills needed in a knowledge based society;<\/li>\n<li>assessment of the higher levels of learning remains a challenge for MOOCs, to the extent that most MOOC providers will not recognise their own MOOCs for credit;<\/li>\n<li>MOOC materials may be limited by copyright or time restrictions for re-use as open educational resources.<\/li>\n<\/ul>\n<div class=\"bcc-box bcc-info\">\n<h3 itemprop=\"educationalUse\">Activity 5.4\u00a0Assessing the strengths and weaknesses of MOOCs<\/h3>\n<p>1. Do you agree that MOOCs are just another form of educational broadcasting? What are your reasons?<\/p>\n<p><span>2. Is it reasonable to compare the costs of xMOOCs to the costs of online credit courses? Are they competing for the same funds, or are they categorically different in their funding source and goals? If so, how? <\/span><\/p>\n<p><span>3. Could you make the case that cMOOCs are a better value proposition than xMOOCs \u2013 or are they again too different to compare?\u00a0<\/span><\/p>\n<p>4. MOOCs are clearly cheaper than either face-to-face or online credit courses if judged on the cost per participant successfully completing a course. Is this a fair comparison, and if not, why not?<\/p>\n<p>5. Do you think institutions should give credit for students successfully completing MOOCs? If so, why, and what are the implications?<\/p>\n<\/div>\n<h1>References<\/h1>\n<p><span>Balfour, S. P. (2013)\u00a0<a href=\"Automated essay scoring and calibrated peer review. Research &amp; Practice in Assessment, 8.\" style=\"font-style: italic\">Assessing writing in MOOCs: Automated essay scoring and calibrated peer review<\/a>\u00a0<\/span><em>Research &amp; Practice in Assessment, <\/em>Vol. 8.<\/p>\n<p>Bates, A. (1985) <em>Broadcasting in Education: An Evaluation<\/em> London: Constables<\/p>\n<p>Bates, A. and Sangr\u00e0, A. (2011) <em>Managing Technology in Higher Education<\/em> San Francisco: Jossey-Bass\/John Wiley and Co<\/p>\n<p>Bayne, S. (2014) <em>Teaching, Research and the More-than-Human in Digital Education<\/em>\u00a0Oxford UK:\u00a0EDEN Research Workshop (keynote: no printed record available)<\/p>\n<p>Book, P. (2103)\u00a0<em>ACE as Academic Credit Reviewer\u2013Adjustment, Accommodation, and Acceptance<\/em> <a href=\"http:\/\/wcetblog.wordpress.com\/2013\/07\/25\/ace-review-moocs-for-credit\/\">WCET Learn<\/a>, July 25<\/p>\n<p>Colvin, K. et al. (2014) <em><a href=\"http:\/\/www.tonybates.ca\/2014\/10\/19\/what-students-learned-from-an-mit-physics-mooc\/#sthash.zqWeKuRA.dpuf\">Learning an Introductory Physics MOOC: All Cohorts Learn Equally, Including On-Campus Class<\/a><\/em>, IRRODL, Vol. 15, No. 4<\/p>\n<p>Dillenbourg, P. (ed.) (1999)\u00a0<em>Collaborative-learning: Cognitive and Computational Approaches<\/em>. Oxford: Elsevier<\/p>\n<p>Dillenbourg, P. (2014) <em>MOOCs: Two Years Later<\/em>, Oxford UK:\u00a0EDEN Research Workshop\u00a0<span>(keynote: no printed record available)<\/span><\/p>\n<p>Engle, W. (2104) <a href=\"http:\/\/flexible.learning.ubc.ca\/files\/2014\/09\/MOOC-Report.pdf\"><em>UBC MOOC Pilot: Design and Delivery<\/em><\/a> Vancouver BC: University of British Columbia<\/p>\n<p>Falchikov, N. and Goldfinch, J. (2000) <em>Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks<\/em> <a href=\"http:\/\/rer.sagepub.com\/content\/70\/3\/287.short\">Review of Educational Research<\/a>, Vol. 70, No. 3<\/p>\n<p>Firmin, R. et al. (2014) Case study: using MOOCs for conventional college coursework\u00a0<em>Distance Education<\/em>, Vol. 35, No. 2<\/p>\n<p>Harasim, L. (2012)\u00a0<em>Learning Theory and Online Technologies<\/em>\u00a0New York\/London: Routledge<\/p>\n<p>Haynie, D. (2014). <a href=\"http:\/\/www.usnews.com\/education\/online-education\/articles\/2014\/01\/20\/state-department-hosts-mooc-camp-for-online-learners-abroad\">State Department hosts &#8216;MOOC Camp&#8217; for online learners<\/a>. US News,January 20<\/p>\n<p>Hill, P. (2013) <a href=\"http:\/\/mfeldstein.com\/validation-mooc-student-patterns-graphic\/\">Some validation of MOOC student patterns graphic<\/a>, e-Literate, August 30<\/p>\n<p><span>Ho, A. et al. (2014)\u00a0<\/span><a href=\"http:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=2381263\">HarvardX and MITx: The First Year of Open Online Courses Fall 2012-Summer 2013\u00a0<\/a><span>(HarvardX and MITx Working Paper No. 1), January 21\u00a0<\/span><\/p>\n<p><span>Hollands, F. and Tirthali, D. (2014) <\/span><a href=\"http:\/\/cbcse.org\/wordpress\/wp-content\/uploads\/2014\/05\/MOOCs_Expectations_and_Reality.pdf\">MOOCs: Expectations and Reality<\/a><span> New York: Columbia University Teachers\u2019 College, Center for Benefit-Cost Studies of Education<\/span><\/p>\n<p>H\u00fclsmann, T. (2003) Costs without camouflage: a cost analysis of Oldenburg University&#8217;s \u00a0two graduate certificate programs offered \u00a0as part of the online Master of Distance Education (MDE): a case study, in Bernath, U. and Rubin, E., (eds.) <em>Reflections on Teaching in an Online Program: A Case Study<\/em> Oldenburg, Germany: Bibliothecks-und Informationssystem der Carl von Ossietsky Universit\u00e4t Oldenburg<\/p>\n<p>Jaschik, S. (2013) MOOC Mess, <a href=\"https:\/\/www.insidehighered.com\/news\/2013\/02\/04\/coursera-forced-call-mooc-amid-complaints-about-course\">Inside Higher Education<\/a>, February 4<\/p>\n<p>Knox, J. (2014) Digital culture clash: &#8216;massive&#8217; education in the e-Learning and Digital Cultures\u00a0<em>Distance Education<\/em>, Vol. 35, No. 2<\/p>\n<p class=\"a-size-large a-spacing-none\">Kop, R. (2011) The Challenges to Connectivist Learning on Open Online Networks: Learning Experiences during a Massive Open Online Course\u00a0<em><a href=\"http:\/\/www.irrodl.org\/index.php\/irrodl\/article\/view\/882\/1823\">International\u00a0Review of Research into Open and Distance Learning<\/a><\/em>, Vol. 12, No. 3<\/p>\n<p class=\"a-size-large a-spacing-none\">Lave, J. and<span style=\"color: #252525\">\u00a0<\/span>Wenger, E.<span style=\"color: #252525\">\u00a0(1991).\u00a0<\/span><a class=\"external text\" href=\"http:\/\/books.google.com\/?id=CAVIOrW3vYAC\" rel=\"nofollow\" style=\"color: #663366\"><i>Situated Learning: Legitimate Peripheral Participation<\/i><\/a><span style=\"color: #252525\">. Cambridge: Cambridge University Press<\/span><\/p>\n<p><span>Milligan, C., Littlejohn, A. and Margaryan, A. (2013) Patterns of engagement in connectivist MOOCs, <\/span><em><a href=\"http:\/\/jolt.merlot.org\/vol9no2\/milligan_0613.htm\">Merlot Journal of Online Learning and Teaching<\/a><\/em><span>, Vol. 9, No. 2<\/span><\/p>\n<p>Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., &amp; Koller, D. (2013)\u00a0<a href=\"http:\/\/www.stanford.edu\/~cpiech\/bio\/papers\/tuningPeerGrading.pdf\"><em>Tuned models of peer assessment in MOOCs<\/em><\/a>. Palo Alto, CA: Stanford University<\/p>\n<p>Rumble, G. (2001) The costs and costing of networked learning, <a href=\"http:\/\/www.c3l.uni-oldenburg.de\/cde\/media\/readings\/rumb01d.pdf\">Journal of Asynchronous Learning Networks<\/a>, Vol. 5, No. 2<\/p>\n<p class=\"a-size-large a-spacing-none\">Suen, H. (2104) <em>Peer assessment for massive open online courses (MOOCs) <a href=\"http:\/\/www.irrodl.org\/index.php\/irrodl\/article\/view\/1680\/2904\">International\u00a0Review of Research into Open and Distance Learning<\/a><\/em>, Vol. 15, No. 3<\/p>\n<p class=\"a-size-large a-spacing-none\"><span>University of Ottawa (2013) <\/span><a href=\"http:\/\/www.uottawa.ca\/vr-etudes-academic\/en\/documents\/e-learning-working-group-report.pdf\">Report of the e-Learning Working Group<\/a><span> Ottawa ON: The University of Ottawa<\/span><\/p>\n<p class=\"a-size-large a-spacing-none\">van Zundert, M., Sluijsmans, D., van Merri\u00ebnboer, J. (2010). Effective peer assessment processes: Research findings and future directions. <em>Learning and Instruction, 20,<\/em> 270-279<\/p>\n","protected":false},"author":30,"menu_order":4,"template":"","meta":{"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":[],"pb_section_license":""},"chapter-type":[],"contributor":[],"license":[],"class_list":["post-826","chapter","type-chapter","status-publish","hentry"],"part":812,"_links":{"self":[{"href":"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-json\/pressbooks\/v2\/chapters\/826","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-json\/wp\/v2\/users\/30"}],"version-history":[{"count":25,"href":"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-json\/pressbooks\/v2\/chapters\/826\/revisions"}],"predecessor-version":[{"id":4026,"href":"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-json\/pressbooks\/v2\/chapters\/826\/revisions\/4026"}],"part":[{"href":"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-json\/pressbooks\/v2\/parts\/812"}],"metadata":[{"href":"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-json\/pressbooks\/v2\/chapters\/826\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-json\/wp\/v2\/media?parent=826"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-json\/pressbooks\/v2\/chapter-type?post=826"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-json\/wp\/v2\/contributor?post=826"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/opentextbc.ca\/teachinginadigitalage\/wp-json\/wp\/v2\/license?post=826"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}