Dallas Baptist University
Online Education Program
Best Practices for Electronically Offered Degree and Certificate Programs
July 2007

5. Evaluation and Assessment

Both the assessment of student achievement and evaluation of the overall program take on added importance as new techniques evolve. For example, in asynchronous programs the element of seat time is essentially removed from the equation. For these reasons, the institution conducts sustained, evidence-based and participatory inquiry as to whether distance learning programs are achieving objectives. The results of such inquiry are used to guide curriculum design and delivery, pedagogy, and educational processes, and may affect future policy and budgets perhaps have implications for the institution’s roles and mission.

Dallas Baptist University employs sound and acceptable practices for determining the amount and level of credit awarded for all courses, regardless of format or mode of delivery. DBU online and hybrid courses are adapted from existing traditional courses and receive the same credit as their traditional counterparts. Credit for courses taken through consortia relationships with DBU is addressed in Comprehensive Standard 3.4.7. The Institution awards reasonable credit for a course by instituting stringent processes to determine that the course is creditable according to the high standards of the institution, and by demonstrating understanding of the class length and level of rigor. Faculty, Chairs, Deans, Provost, and the Vice President for Graduate and Corporate Affairs are involved in the different levels for the approvals and evaluations of all coursework.

DBU online and hybrid courses are developed using the same course objectives and similar learning outcomes and are assessed using multiple forms of course assessment such as online exams, essays, research papers, graded class discussion, groupwork, case studies, and simulated exercises.  All practices for determining the amount and level of credit awarded for courses are the same for all courses regardless of format or mode of delivery. Curricula in every major academic field are evaluated on an on-going basis by faculty teaching in that field through a comprehensive system of planning and evaluation. These evaluations provide a benchmark for an annual self-assessment required of every Dallas Baptist University college and educational program. Consistent efforts are made to ensure that all areas of curriculum are appropriate, relevant, well-coordinated, and supervised by well-qualified faculty.

All courses, regardless of format or delivery, go through a standard process of approval. When the course proposal is first submitted, the semester credit hour value for the course is also proposed and undergoes review during the entire approval process. Course development at Dallas Baptist University is faculty driven. The persons with academic credentials in the field are knowledgeable and closest to the processes of curricular development and review.

DBU faculty have access to an online gradebook provided in the courseware management system, Blackboard, used to support teaching and learning in both traditional and online classes. The course gradebook is used to calculate and display grades and provides clear feedback on assessment using online quizzes and exams, journals, and assignment tools, in which the student can check with a simple click from the course menu of each course home page. The student only has access to his/her grades and can click on the actual grade to view instructor comments on their class assignments.

Instructors access their course gradebooks from their course control panel. The gradebook is similar to an Excel spreadsheet with columns for each grade and rows for each student, with each row providing a total of all grade points earned. The grades can be categorized, weights applied, and easily sorted, and downloaded to Excel if necessary. For those instructors using Turnitin.com to compare documents for plagiarism, the gradebook provides an easy to use format for viewing the document report while assessing the grade.

The newest version of Blackboard (7.3) that will be installed for the Fall 2007 semester has improved this tool by offering an averaging capability, instructor feedback areas for all assessments, and an early warning system to instructors notifying them of at-risk students that fall below an instructor established range of grades. Through these technological advances, college level competencies may be more effectively assessed and data utilized to better guide learning.

An initial comparison of online and hybrid course grades to traditional course grades was performed for Fall 2006 and Spring 2007 that reveals grades are closely comparable to those in the traditional classroom, with an estimated .25 increase/decrease in grade points either way.

A longer term general studies assessment strategy is under development to better enable faculty to measure and improve overall student learning, utilizing data gleaned from written examinations, essays, papers, and presentations which contain common course outcome evaluation in all sections of any general studies course. This data will be aggregated for each discipline regardless of delivery method to assess overall learner outcomes in general studies courses.

All online courses are developed with assessments that may include exams, either paper-based in a take home format or timed, online delivery. Two courses do require proctored examinations and these are noted in the class schedule as requiring proctored exams: ENGL 1301 Introduction to Language and Literature I and MANA 3305 Managerial Statistics. Online students are responsible for securing a qualified proctor and supplying the proctor's name, position/title, email address, and phone and fax numbers to the instructor. A qualified proctor includes a minister or church staff member; a librarian at a local library (excluding DBU's librarians) or a supervisor or Human Resource Department staff at their place of employment. The proctor will provide a faxed copy of the student's identification to the instructor. Students receive more detailed instructions regarding proctor guidelines and procedures.

While some online examinations are used as student assessment, procedures are in place to deter cheating. Most online exams have double and triple the amount of questions necessary so that each student receives a different exam that is randomly chosen from the exam bank. Additionally, instructors teaching online are encouraged to develop courses that assess beyond objective testing to include essays, case studies, reflective exercises, critical thinking exercises, and research papers so that assessment becomes more like a portfolio or overall snapshot of achievement for each course. No online course completely relies upon online exams for assessment. Most courses use online exams for 30-35% of the overall course assessment for grading.

DBU delivers online course materials through Blackboard, a leader in online courseware management systems (see Appendix 26). The course management system provides tools in which instructors can place students' grades and feedback within a protected and secure environment. Students must log into Blackboard using their secure DBU Student login and password through an encrypted access point. Students may access only their assessment grades through Blackboard and their final course grades through the very secure DBU Webadvisor system.

All personal information for both students and instructors in Blackboard is protected and encrypted through an SSL security certificate process and social security numbers are not transmitted into the Blackboard system but reside only in the student information system, Datatel. There is no known incident of Blackboard having been breached at any other institution.

Student course/faculty evaluations are also delivered through a secure system within Blackboard that allows students to complete evaluations anonymously and protect information provided.

5d. Overall program effectiveness is determined by such measures as: The extent to which student learning matches intended outcomes, including for degree programs both the goals of general education and the objectives of the major. The extent to which student intent is met. Student retention rates, including variations over time. Student satisfaction, as measured by regular surveys. Faculty satisfaction, as measured by regular surveys and by formal and informal peer review processes. The extent to which access is provided to students not previously served.  Measures of the extent to which library and learning resources are used appropriately by the program’s students. Measures of student competence in fundamental skills such as communication, comprehension, and analysis. Cost effectiveness of the program to its students, as compared to campus-based alternatives.

In support of DBU’s commitment to institutional effectiveness, the Online Education Department has developed and implemented an assessment plan. The primary evaluation techniques used to assess institutional effectiveness in the Online Education Department include:

1. Comparative Report. The Comparative Report tracks enrollment from one semester to the next, details potential and actual enrollment percentages, notes completion rates, and provides demographic information regarding online students. A copy of this report is included in Appendix 53b.
   
   
2. Online Student Skills Assessment. The Online Student Skills Assessment is used to evaluate the computer and Internet skills of prospective online students. A copy of this assessment is included in Appendix 29.
   
   
3. Student/Faculty Course Evaluation Form. The Student/Faculty Course Evaluation Form measures student satisfaction and faculty development. In 2002-2003, the DBU Webmaster created an in-house system for Student/Faculty Course Evaluation delivered via email and the Internet. This form is included in Appendix 49. Beginning with the fall 2004 semester, the Computing and Technology department improved the Student/Faculty evaluation process by creating a plug-in tool that deploys the evaluations directly through Blackboard without faculty or students having access to the results. The data is gathered electronically and provided to the Institutional Research department for compilation and reporting. This tool allows students to access the evaluation right from their online course and post completely anonymous feedback.
   
   
4. Hybrid: Regular Comparative Report. The Hybrid Regular Comparative Report examines student satisfaction in both hybrid and regular classroom courses taught by the same instructor. This data is included in Appendix 53d.
   
   
5. Student Satisfaction Survey each spring (Appendix 53e).
   
   
6. Grade Comparison Chart for courses that are offered online/hybrid and traditional. The initial report compared Fall 2006 and Spring 2007 classes to show that grades received in online/hybrid classes are closely comparable to those in the traditional classroom, with an estimated .25 increase/decrease in grade points either way.

The Online Education Department, under the direction of the Dean and the Office of Provost, identifies key indicators to monitor the success and continued improvement of the online programs. The Online Education Department collects and analyzes data to report assessment results and trends for online programs.

In 2006, the DBU Online Education program entered into a self-audit to benchmark against other quality online programs. This opportunity was provided by the AHEC – Alliance for Higher Education Competitiveness. The graphic below demonstrates the six key factors audited in the process. DBU Online Education was ranked as a Best Practice Leader. See Appendix 51 for the complete self-audit report.

Additionally, the Online Education Department completes several major assignments that provide benchmark data for program assessment. Some of the primary activities are identified in the following table:

Whether programs are offered online or in the classroom, DBU considers quality teaching to be its primary focus. Scholarship and public service are critical elements in the development and maintenance of quality teaching. Accordingly, the university engages in an ongoing process of planning and evaluation of all three activities to ensure the quality of student learning. Systematic, broad-based, interrelated planning and evaluation support DBU's educational activities through several important means.

Educational effectiveness and the quality of student learning are assessed at both the college and major level by the faculty involved in the administration of each program. Faculty members' professional development fosters improvements in course design which leads to more effective teaching/learning. This process is evidenced through the Program Description and Annual Report that is developed each year for program assessment and improvement. In fact, all academic divisions have been involved in the Annual Report and Program Description process used to facilitate the strengthening of academic programs since 1997. Annual administrative program description documents and educational support program description documents detail each department's comprehensive system for planning and evaluation.

Each year, every educational unit and support service is required to present a self-analysis which includes a Program Description and Annual Report. Academic and academic support units are trained to think critically about institutional effectiveness. In addition to historical reference data and notable events for the prior year, the report details self-determined departmental goals and learner-centered objectives, the metric for evaluating the success rate of each goal, and the improvements that were instituted in response. These are submitted to the Office of Institutional Effectiveness during a celebratory event known as the Ingathering. The supervising Vice President appraises the progress of the department. These documents are filed in the Office of Institutional Effectiveness. Annual reports identify goals and data and progress toward goals and adjustments made to improve each program by reviewing the prior year's goals based on a set of defined measurable criteria and determine what changes should be made based on the data.

Educational effectiveness and the quality of student learning in every college and major are also evaluated by the Institutional Effectiveness Committee throughout a 10-year cycle, again, without respect to online or classroom delivery as all programs are evaluated. As data reveals the need for adjustment in curriculum and/or methodology, other evaluative groups assist in the evaluation of data and program development (e.g., the Faculty Curriculum Committee, Faculty Council, Distance Education Steering Committee, and various ad hoc study committees that are appointed as needed).

(Adapted from DBU's SACS Compliance Certification, 2.5 and 3.3.1)

Programs and Academic Descriptions >

Organizational Chart >

Programs Descriptions and Annual Reports >

Organizational Chart >

Review of Distance Learning Programs >