{"id":636,"date":"2019-02-13T19:44:18","date_gmt":"2019-02-13T19:44:18","guid":{"rendered":"https:\/\/opentextbc.ca\/workinggroupguide\/?post_type=chapter&#038;p=636"},"modified":"2022-09-15T17:58:28","modified_gmt":"2022-09-15T17:58:28","slug":"evaluate","status":"publish","type":"chapter","link":"https:\/\/opentextbc.ca\/workinggroupguide\/chapter\/evaluate\/","title":{"raw":"Evaluate","rendered":"Evaluate"},"content":{"raw":"<blockquote>Evaluation is the collection of, analysis and interpretation of information about any aspect of a programme of education or training as part of a recognized process of judging its effectiveness, its efficiency and any other outcomes it may have.[footnote]Fred Percival, Henry Ellington, and Phil Race, <em>A Handbook of Educational Technology<\/em>, 3rd Edition <em>(<\/em>London:\u00a0Kogan Page Ltd., 1993).[\/footnote]\r\n\r\nMary Thorpe<\/blockquote>\r\n<h1>How are evaluation and sustainability related?<\/h1>\r\nEvaluating whether your open working group has met its goals and stated outcomes is an essential part of ensuring its sustainability. (See <a class=\"internal\" href=\"\/workinggroupguide\/chapter\/goals-and-purpose\/\">Goals and Purpose<\/a>.) Communicating to stakeholders\u2014including senior administration and student groups\u2014about success and impacts of activities such as events, grants, or program\/policy changes may lead to increased support and further initiatives. However, evaluating the impact of OER adoption and advocacy can be challenging, so it will be important to consider the type of tools to use and evidence to gather early when establishing your open working group:\r\n<blockquote>\u2026evidence generated by complex and innovative processes such as OER release is often itself complex, context-specific and difficult to generalise. These kinds of initiatives require significant organisational change and may include external partners and stakeholders with very different cultures and practices. Evaluation, in particular is challenging and ranges from evaluating specific OER for fitness of purpose, changes in staff attitudes, impact on learning and teaching and longer term impact on institutional practices and the wider community.\"[footnote]Lou McGill, \"Open Educational Resources: Sustainability,\" Jisc,\u00a0https:\/\/www.jisc.ac.uk\/guides\/open-educational-resources\/sustainability\u00a0(accessed\u00a0January 25, 2019).[\/footnote]<\/blockquote>\r\n<div class=\"textbox textbox--exercises\"><header class=\"textbox__header\">\r\n<p class=\"textbox__title\">Questions to Consider: About Evaluation &amp; Sustainability<\/p>\r\n\r\n<\/header>\r\n<div>\r\n<ul>\r\n \t<li>What are we evaluating?<\/li>\r\n \t<li>How do you build in evaluation?<\/li>\r\n \t<li>What is the open working group's goal in evaluation?<\/li>\r\n \t<li>How do you report?<\/li>\r\n \t<li>How do report your own activity (e.g. advocacy, capacity building, awareness)?<\/li>\r\n \t<li>How do we report on and show value for the working group itself?<\/li>\r\n<\/ul>\r\n<\/div>\r\n<\/div>\r\n<h1>Why do we evaluate?<\/h1>\r\nThere are many reasons why open working groups should evaluate their work. It allows you to:\r\n<ul>\r\n \t<li>Share cost savings with students, governments, and institutions. This can provide a rationale for government and institutional funding and student buy-in.<\/li>\r\n \t<li>Link adoption of resources with student learning (i.e., students contributing to resources, customization by faculty).<\/li>\r\n \t<li>Determine and ensure OER quality. Quality is a central concern for faculty in considering OER adoption.[footnote]I. Elaine Allen and Jeff Seaman, \u201cOnline Report Card: Tracking Online Education in the United States,\u201d <em>Babson Survey Research Group<\/em>,\u00a02016.[\/footnote]<\/li>\r\n \t<li>Share success and OER adoption. Sharing adoptions and the success of open education programs and projects encourages adoptions and increases the awareness of OER.<\/li>\r\n<\/ul>\r\n<h1>Who evaluates?<\/h1>\r\nDifferent institutional stakeholders often are charged with evaluating OER:\r\n<ul>\r\n \t<li><strong>Students<\/strong>. For example, <a href=\"https:\/\/cuny.manifoldapp.org\/system\/resource\/69eae5ed-33b7-4a01-bb22-ee49b92d7299\/attachment\/original-0d1a92ee2b3110b8e953c1a0fc71c3eb.pdf\">students at UBC led an environmental scan of OER [PDF]<\/a>.<\/li>\r\n \t<li><strong>Faculty researching OER<\/strong>. For example,\u00a0<a href=\"https:\/\/www.researchgate.net\/profile\/Arthur_Green\">Gill Green at UBCO<\/a>,\u00a0<a href=\"https:\/\/www.researchgate.net\/scientific-contributions\/2135396649_Christina_Hendricks\">Christina Hendricks at UBC<\/a>,\u00a0<a href=\"https:\/\/www.researchgate.net\/scientific-contributions\/2095559188_Rajiv_S_Jhangiani\">Rajiv Jhangiani at KPU<\/a>, and <a href=\"https:\/\/www.researchgate.net\/profile\/Georg-Rieger\">Georg Rieger at UBC<\/a>.<\/li>\r\n \t<li><strong>Libraries<\/strong>. For example, this research paper on a community of librarians supporting OER, called <a href=\"https:\/\/bccampus.ca\/wp-content\/uploads\/2016\/04\/DLS-Conference.pdf\">Librarians and OER: Cultivating a Community of Practice to be more Effective Advocates [PDF]<\/a>.<\/li>\r\n \t<li><strong>Instructional support staff researching OER<\/strong>.<\/li>\r\n \t<li><strong>Centres for teaching and learning<\/strong>.<\/li>\r\n<\/ul>\r\n<div class=\"textbox textbox--examples\"><header class=\"textbox__header\">\r\n<p class=\"textbox__title\">In Practice<\/p>\r\n\r\n<\/header>\r\n<div class=\"textbox__content\">\r\n\r\nThe TRU Special Investment Fund grant, which supports faculty development, has the following requirements built in. The open working group must collect and report on the following:\r\n<ul>\r\n \t<li>Total investment in grants programs,<\/li>\r\n \t<li>Number of funded courses,<\/li>\r\n \t<li>Total enrolment in funded courses over time,<\/li>\r\n \t<li>Average textbook costs (before OER development),<\/li>\r\n \t<li>Total student savings after one year, and<\/li>\r\n \t<li>The extent of faculty dissemination of their work in developing and integrating OER.<\/li>\r\n<\/ul>\r\n<\/div>\r\n<\/div>","rendered":"<blockquote><p>Evaluation is the collection of, analysis and interpretation of information about any aspect of a programme of education or training as part of a recognized process of judging its effectiveness, its efficiency and any other outcomes it may have.<a class=\"footnote\" title=\"Fred Percival, Henry Ellington, and Phil Race, A Handbook of Educational Technology, 3rd Edition (London:\u00a0Kogan Page Ltd., 1993).\" id=\"return-footnote-636-1\" href=\"#footnote-636-1\" aria-label=\"Footnote 1\"><sup class=\"footnote\">[1]<\/sup><\/a><\/p>\n<p>Mary Thorpe<\/p><\/blockquote>\n<h1>How are evaluation and sustainability related?<\/h1>\n<p>Evaluating whether your open working group has met its goals and stated outcomes is an essential part of ensuring its sustainability. (See <a class=\"internal\" href=\"\/workinggroupguide\/chapter\/goals-and-purpose\/\">Goals and Purpose<\/a>.) Communicating to stakeholders\u2014including senior administration and student groups\u2014about success and impacts of activities such as events, grants, or program\/policy changes may lead to increased support and further initiatives. However, evaluating the impact of OER adoption and advocacy can be challenging, so it will be important to consider the type of tools to use and evidence to gather early when establishing your open working group:<\/p>\n<blockquote><p>\u2026evidence generated by complex and innovative processes such as OER release is often itself complex, context-specific and difficult to generalise. These kinds of initiatives require significant organisational change and may include external partners and stakeholders with very different cultures and practices. Evaluation, in particular is challenging and ranges from evaluating specific OER for fitness of purpose, changes in staff attitudes, impact on learning and teaching and longer term impact on institutional practices and the wider community.&#8221;<a class=\"footnote\" title=\"Lou McGill, &quot;Open Educational Resources: Sustainability,&quot; Jisc,\u00a0https:\/\/www.jisc.ac.uk\/guides\/open-educational-resources\/sustainability\u00a0(accessed\u00a0January 25, 2019).\" id=\"return-footnote-636-2\" href=\"#footnote-636-2\" aria-label=\"Footnote 2\"><sup class=\"footnote\">[2]<\/sup><\/a><\/p><\/blockquote>\n<div class=\"textbox textbox--exercises\">\n<header class=\"textbox__header\">\n<p class=\"textbox__title\">Questions to Consider: About Evaluation &amp; Sustainability<\/p>\n<\/header>\n<div>\n<ul>\n<li>What are we evaluating?<\/li>\n<li>How do you build in evaluation?<\/li>\n<li>What is the open working group&#8217;s goal in evaluation?<\/li>\n<li>How do you report?<\/li>\n<li>How do report your own activity (e.g. advocacy, capacity building, awareness)?<\/li>\n<li>How do we report on and show value for the working group itself?<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<h1>Why do we evaluate?<\/h1>\n<p>There are many reasons why open working groups should evaluate their work. It allows you to:<\/p>\n<ul>\n<li>Share cost savings with students, governments, and institutions. This can provide a rationale for government and institutional funding and student buy-in.<\/li>\n<li>Link adoption of resources with student learning (i.e., students contributing to resources, customization by faculty).<\/li>\n<li>Determine and ensure OER quality. Quality is a central concern for faculty in considering OER adoption.<a class=\"footnote\" title=\"I. Elaine Allen and Jeff Seaman, \u201cOnline Report Card: Tracking Online Education in the United States,\u201d Babson Survey Research Group,\u00a02016.\" id=\"return-footnote-636-3\" href=\"#footnote-636-3\" aria-label=\"Footnote 3\"><sup class=\"footnote\">[3]<\/sup><\/a><\/li>\n<li>Share success and OER adoption. Sharing adoptions and the success of open education programs and projects encourages adoptions and increases the awareness of OER.<\/li>\n<\/ul>\n<h1>Who evaluates?<\/h1>\n<p>Different institutional stakeholders often are charged with evaluating OER:<\/p>\n<ul>\n<li><strong>Students<\/strong>. For example, <a href=\"https:\/\/cuny.manifoldapp.org\/system\/resource\/69eae5ed-33b7-4a01-bb22-ee49b92d7299\/attachment\/original-0d1a92ee2b3110b8e953c1a0fc71c3eb.pdf\">students at UBC led an environmental scan of OER [PDF]<\/a>.<\/li>\n<li><strong>Faculty researching OER<\/strong>. For example,\u00a0<a href=\"https:\/\/www.researchgate.net\/profile\/Arthur_Green\">Gill Green at UBCO<\/a>,\u00a0<a href=\"https:\/\/www.researchgate.net\/scientific-contributions\/2135396649_Christina_Hendricks\">Christina Hendricks at UBC<\/a>,\u00a0<a href=\"https:\/\/www.researchgate.net\/scientific-contributions\/2095559188_Rajiv_S_Jhangiani\">Rajiv Jhangiani at KPU<\/a>, and <a href=\"https:\/\/www.researchgate.net\/profile\/Georg-Rieger\">Georg Rieger at UBC<\/a>.<\/li>\n<li><strong>Libraries<\/strong>. For example, this research paper on a community of librarians supporting OER, called <a href=\"https:\/\/bccampus.ca\/wp-content\/uploads\/2016\/04\/DLS-Conference.pdf\">Librarians and OER: Cultivating a Community of Practice to be more Effective Advocates [PDF]<\/a>.<\/li>\n<li><strong>Instructional support staff researching OER<\/strong>.<\/li>\n<li><strong>Centres for teaching and learning<\/strong>.<\/li>\n<\/ul>\n<div class=\"textbox textbox--examples\">\n<header class=\"textbox__header\">\n<p class=\"textbox__title\">In Practice<\/p>\n<\/header>\n<div class=\"textbox__content\">\n<p>The TRU Special Investment Fund grant, which supports faculty development, has the following requirements built in. The open working group must collect and report on the following:<\/p>\n<ul>\n<li>Total investment in grants programs,<\/li>\n<li>Number of funded courses,<\/li>\n<li>Total enrolment in funded courses over time,<\/li>\n<li>Average textbook costs (before OER development),<\/li>\n<li>Total student savings after one year, and<\/li>\n<li>The extent of faculty dissemination of their work in developing and integrating OER.<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<hr class=\"before-footnotes clear\" \/><div class=\"footnotes\"><ol><li id=\"footnote-636-1\">Fred Percival, Henry Ellington, and Phil Race, <em>A Handbook of Educational Technology<\/em>, 3rd Edition <em>(<\/em>London:\u00a0Kogan Page Ltd., 1993). <a href=\"#return-footnote-636-1\" class=\"return-footnote\" aria-label=\"Return to footnote 1\">&crarr;<\/a><\/li><li id=\"footnote-636-2\">Lou McGill, \"Open Educational Resources: Sustainability,\" Jisc,\u00a0https:\/\/www.jisc.ac.uk\/guides\/open-educational-resources\/sustainability\u00a0(accessed\u00a0January 25, 2019). <a href=\"#return-footnote-636-2\" class=\"return-footnote\" aria-label=\"Return to footnote 2\">&crarr;<\/a><\/li><li id=\"footnote-636-3\">I. Elaine Allen and Jeff Seaman, \u201cOnline Report Card: Tracking Online Education in the United States,\u201d <em>Babson Survey Research Group<\/em>,\u00a02016. <a href=\"#return-footnote-636-3\" class=\"return-footnote\" aria-label=\"Return to footnote 3\">&crarr;<\/a><\/li><\/ol><\/div>","protected":false},"author":90,"menu_order":3,"template":"","meta":{"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":[],"pb_section_license":""},"chapter-type":[],"contributor":[],"license":[],"class_list":["post-636","chapter","type-chapter","status-publish","hentry"],"part":24,"_links":{"self":[{"href":"https:\/\/opentextbc.ca\/workinggroupguide\/wp-json\/pressbooks\/v2\/chapters\/636","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/opentextbc.ca\/workinggroupguide\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/opentextbc.ca\/workinggroupguide\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/opentextbc.ca\/workinggroupguide\/wp-json\/wp\/v2\/users\/90"}],"version-history":[{"count":8,"href":"https:\/\/opentextbc.ca\/workinggroupguide\/wp-json\/pressbooks\/v2\/chapters\/636\/revisions"}],"predecessor-version":[{"id":862,"href":"https:\/\/opentextbc.ca\/workinggroupguide\/wp-json\/pressbooks\/v2\/chapters\/636\/revisions\/862"}],"part":[{"href":"https:\/\/opentextbc.ca\/workinggroupguide\/wp-json\/pressbooks\/v2\/parts\/24"}],"metadata":[{"href":"https:\/\/opentextbc.ca\/workinggroupguide\/wp-json\/pressbooks\/v2\/chapters\/636\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/opentextbc.ca\/workinggroupguide\/wp-json\/wp\/v2\/media?parent=636"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/opentextbc.ca\/workinggroupguide\/wp-json\/pressbooks\/v2\/chapter-type?post=636"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/opentextbc.ca\/workinggroupguide\/wp-json\/wp\/v2\/contributor?post=636"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/opentextbc.ca\/workinggroupguide\/wp-json\/wp\/v2\/license?post=636"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}