Using LMS Data to Inform Course Design
What
What is this resource about? Learning Management Systems (LMSs) that support online and blended courses provide data on students’ actions online. These data can be paired with data on student achievement, persistence, or additional qualitative data to support strategic decision making in iterative instructional design resulting in increasingly effective instructional environments that address learner variability and engage all students. In this resource, watch an Example, learn more about Types of Data, Analysis of Data, the Emphasis to Improve the Learning Enviornment, and the Interpretation of Data through the UDL Principles. Read about the Future Directions and see more under Resources.
Why
Why is this important for higher education? An iterative relationship between data analysis and course design results in refined efforts at addressing learner variability and engaging all students – and, as such, increasingly effective instructional environments.
UDL Connection
A UDL approach to data analytics includes three key components:
- emphasis on data to improve the learning environment rather than an individual student
- interpretation of data through the lens of the three UDL principles
- use of data to take action to make improvements to a course
Example
This video describes the process of designing courses to utilize learning management system data.
Types of Data
The types of data that are most widely available in Learning Management Systems (LMSs) can be classified into three broad categories: demographic data, usage data, and achievement data.
- Demographic data: Information about students such as enrollment status, selected majors and courses of study, cumulative GPA, prior degree completion, etc. If this data is not available in the LMS, it is most likely housed in an institution’s online Student Information System (SIS).
- Usage data: Also known sometimes as “click-by-click” data, this information reveals how students navigate through and use the features of an online course (e.g., login/logout times; dwell-time on activities such as a discussion forum; utilization of embedded supports such as videos, text-to-speech, etc.).
- Achievement data: Information about student performance such as quiz and test scores, course and assignment grades, etc.
Analysis of Data
Analysis of the data that are available through an LMS can occur with varying degrees of complexity. To get started, instructors can examine click-by-click data corresponding to online student actions and behaviors in real-time or in the short-term to gain an initial picture of learning environment interaction.
In some LMSs, more extensive analytics can encompass correlations of demographic data with various achievement measures. Aggregated across multiple semesters, or across a department or even an institution, analytics can be a powerful tool to help understand trends in long-term outcomes. For example, an instructor could combine data from students participating in a course over multiple terms in order to find patterns or pathways in enrollment, persistence (or retention), and achievement.
The options for viewing this data will also vary depending on the LMS that is being used. Some software systems generate reports known as “log files” or “event logs” that just have basic data elements such as date of a student’s last login, date of assignment completion, and page views. Other systems provide visualizations of data elements (such as bar graphs) about specific online activity, including student participation in activities and communications between instructors and students. Still other systems integrate demographic information such as student major and year. Some do not have analysis tools readily available. In those cases, if an instructor wants to cross-tabulate, correlate, run more extensive mathematical analyses, or otherwise manage any of the data, the instructor will have to download the data into another software package such as Microsoft Excel.
Emphasis on Data to Improve the Learning Environment
Often times, LMS data appears as a measurement of student abilities: for example, the student scored 75% on a quiz, or the student did not click on all the available resources. UDL encourages instructors to consider that this data actually reflects measurements of the learning environment, not just the student’s abilities. For example, the quiz score is a reflection of how the student expresses knowledge in that format, not just how well the student understands the content; and clicking on certain resources rather than others might be due to the types of resources available (e.g., a PDF of text without vocabulary support vs. an OER with interactive supports for text comprehension) and how accessible they are for the individual student.
Viewed from the UDL angle, LMS data tends to reveal barriers in the environment -- not necessarily what students actually do or do not understand. As instructors increasingly incorporate UDL into their course design, students are increasingly able to comprehend information, engage with it, and act on it in ways that allow them to demonstrate what they know; and the data on their performance and achievement in the course becomes increasingly accurate and actionable. While the use of data to re-design a postsecondary curriculum is relatively uncommon1, the UDL framework can help instructors and course developers use data available through LMSs (or similar systems) to make improvements to online or blended courses that better support the variability present among all students.
Although data is often used in assessing or categorizing students in some capacity, it can also be considered a tool for assessment of the learning environment itself. UDL advocates the importance of utilizing assessment to drive instructional actions in order to improve a learning environment2. More specifically, the type of progress monitoring that can be achieved using the multiple data points provided by online systems is especially consistent with a UDL framework for instruction. This emphasis on what data can reveal about the learning context, rather than limitations of individual students, is a key attribute of the UDL approach to data use and aligns with the emerging field of social learning analytics, which believes that the “focus must not only be on learners, but on their tools and contexts”3.
Interpretation of Data through the UDL Principles
Below, we offer some examples and suggestions as to how to interpret course-level data about student behaviors using the three UDL principles as well as how to translate that information into actionable steps for UDL-aligned iterative course design.
Data on Multiple Means of Representation
Courses can be designed to give instructors information about the content students access and the ways in which these choices relate to other factors such as persistence or achievement.
Data Types
Click-based data describing if, when, and how frequently students are accessing resources that provide varied representations of content (e.g., glossary tools; videos that support background knowledge). Specific data might include:
- Login data that relates to intentionally selected representations of content in a course (e.g., number of students attending synchronous webinars or chats)
- Page-view counts that show how many times students have accessed specific course content
- Activity logs, including which resources are selected
- Forum posts, especially if these utilize various modalities (e.g., video, image, audio, etc.)
Data Use
Questions for Analysis
An instructor can examine the connection between use of different media types and rates of participation and achievement in a course. For example:
- Which core resources are most viewed?
- Which formats of content are most viewed (interactive activities? videos? PDF documents?)
- Is there any relationship between accessing different representations of course content and student achievement?
- Do students who do well in the course tend to access certain resources? What about students who are struggling?
Course Re-design
Data obtained can provide information that guides how an instructor decides to structure their course and illustrate concepts through multiple media. For example, an instructor might:
- Include more tutorial videos if students seem to prefer this type of media or if accessing this type of media is strongly correlated to achievement, or
- Highlight beneficial resources (e.g., an online glossary tool) periodically in communications with students. Consider building these resources into subsequent course units or modules as prominent features if they seem to benefit even a small number of students.
In any of these cases, the design decisions an instructor makes in an attempt to provide options for how students comprehend course content would be increasingly supported by data on what students actually use, and, of those materials, what seems to be the most useful.
Data on Multiple Means of Action and Expression
LMSs provide data that reflect students’ actions and expressions in a course -- such as participation in discussion forums, completion of assignments, and achievement on assessments -- and can be analyzed in conjunction with additional sources of data in order to inform course re-design.
Data Types
Click-based data describing if, when, and how frequently students are participating in discussion forums, completing assignments, and performing on assessments. Specific data might include the following:
- Login dates and times
- Time stamps of when assignments are started and submitted
- Time stamps of when students view pages or course sections with assessment or assignment information
- Participation in and/or attendance in synchronous and/or asynchronous chats
- Grades on assignments
- Data on forum posts and replies
Data Use
Questions for Analysis
By examining data pertaining to student activity in relation to achievement, an instructor can make adjustments to course policies or materials to encourage and support the development of strategic learners. For example:
- Are students uploading assignments or posting to a discussion forum at the last minute just before a deadline, and is this behavior affecting their success in the course?
- If an instructor has introduced multiple options for students to express learning mastery (e.g., e-portfolios, essays, and presentations), which options are students using most often?
- Does the frequency or quality of discussion posts vary across topic or module?
- Does the point at which assessments are introduced make a difference in student achievement? Are these assessments available from the beginning of the course, held back until the end, or released gradually?
Course Re-design
Data available via LMSs can facilitate course re-designs that increasingly reduce barriers in how students express and organize their thinking. For example, if fewer logins or less forum activity are associated with lower achievement, an instructor might:
- Introduce more formative assessments, change assignment guidelines, or initiate more frequent check-ins with students in order to find out what is and is not working, or
- Institute more frequent deadlines and segment assignments into smaller tasks to encourage more frequent logins, or
- Adjust deadlines to mirror participants’ login behavior (e.g., more time between deadlines) and see if this flexibility results in positive achievement.
Information about the point at which students access assignments can be useful for determining when these components of a course ought to be open/presented to students and for analyzing whether any relationship exists between visits to assignments and student achievement. Similarly, data on how students navigate through a course (e.g., what pages students view, what resources they click on, and in what order) and how these pathways align with success in the course can help an instructor decide how to organize materials (e.g., whether material would be best organized chronologically, thematically, or by media type; when is the best time to have students utilize a discussion forum; when assignments should be posted).
Data on Multiple Means of Engagement
The potency of a UDL approach to course design comes from considering all three principles together: examining data with respect to multiple means of representation as well as multiple means of action and expression reveals information about student engagement with the course.
Data Types
Combinations of data that were useful for considering options related to representation and action and expression can also be used to think about student engagement. Other data that would supplement and enrich an instructor’s conceptions of student engagement could include:
- Dwell time (how long a student remained on a page)
- Information on students’ pace through a course (e.g., in a self-paced course, how consistent is a student’s participation or entry into the course?)
- Login data (e.g., frequency, most recent login, attendance during synchronous events such as virtual office hours, web conferences)
- Messaging (Do students message each other? Do students open messages from their instructor? How often does the instructor provide targeted feedback to a group of students, or an individual student? Do students engage in synchronous chatting?)
- Discussion forum activity (e.g., views, posts, replies)
Data Use
Questions for Analysis
The data discussed in the last two sections under multiple means of representation and multiple means of action and expression can often help the instructor think about engagement as well. For example:
- Do students access supports such as peer tutoring, virtual office hours, a glossary, etc.?
- do students take longer to complete a course or its components than expected? If so, is the material appropriately scaffolded? Conversely, if students are proceeding through a course more quickly than the instructor anticipated, are there options for enrichment?
- Are students effectively utilizing choices that are built into the course?
- Does instructor feedback improve student persistance and/or achievement in a course?
Course Re-design
These data can facilitate approaches to course design and decision-making that support engagement amongst variable students. After a thoughtful analysis of these data, an instructor might:
- Consider showing students data that reflect their progress or the general progress of the class as a whole (e.g., showing that overall, students who access certain resources, participate more frequently in discussion forums, etc., tend to perform better on assignments).
- Build in prompts for student self-reflection (through written response, ungraded quizzes, or brief surveys that provide feedback for the instructor and the chance to self-check) that encourage students to reflect on strategies that work best for them.
Similar tools for self-assessment can help students as they work toward goals. The results can complement the instructor’s data analysis and help the instructor gain a deeper understanding of how to interpret and act upon the data.
While an LMS can provide instructors with insight with respect to quantitative data, engagement should also be assessed through more qualitative methods such as the quality of students’ conversations, posts, and assignments. An instructor might also notice if certain topics “trend” in a forum and consider how popular topics could be leveraged to continue to recruit student interest in upcoming units. If an instructor sees that students do not appear to be engaged in the first few weeks of the course, it may be viewed as an opportunity to think about what can make the course more welcoming and make adjustments.
Future Directions
Much of the data that is available to instructors through LMS systems has yet to attain the full potential of the related field of learning analytics, such as the prediction of student behaviors and outcomes and the recommendation of instructional actions. Nonetheless, this resource introduces the concept that even basic data made available to online course instructors can be a powerful tool for inspiring better course design. The first step is to explore the data that are available and build in opportunities for multiple means of representation, action/expression and engagement to enrich and improve existing data collection. Interpretation of this information through the three UDL principles can then drive course designs that are effective for more students. Additional questions for future work in the area of UDL and data on learning environments include the following:
- How can multiple means of expression (e.g., written response vs. diagramming) be analyzed in a comparable and equitable manner, and how can this analysis be brought to scale?
- What kinds of data and how much data should be made available to students, in particular those with self-regulation or executive functioning challenges? Who should have control over LMS data? What is the proper balance between freedom offered to students and control residing with an instructor? How might this dynamic vary in different contexts?
Resources
- Data Wise: Supporting educators in using data to improve learning and teaching
- Penetrating the Fog: Analytics and Learning in Education by George Siemens
- from Martin Cooper's Online Learning Insights blog: How Learning Analytics can Make Instructors More Effective in an Online Course (includes helpful videos about locating LMS data in "Resources: How-To Videos" section)
References
1Méndez, G., Ochoa, X., & Chiluiza, K. (2014, March). Techniques for data-driven curriculum analysis. In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (pp. 148-157). ACM. http://dx.doi.org/10.1145/2567574.2567591
2Shum, S. B. & Ferguson, R. (2012). Social Learning Analytics. Educational Technology & Society, 15(3), 3-26. http://dx.doi.org/10.1145/2330601.2330616
A learning management system is a software application or suite of applications or a web-based system that provides educational programs and their components such as classes, resources, assessment, tools, and communication, etc.; as well as organizational tools for administration, record-keeping, information sharing, database management, etc., with the intention to manage all parts of a learning process.
UDL is an educational approach based on the learning sciences with three primary principles—multiple means of representation of information, multiple means of student action and expression, and multiple means of student engagement.
Video is the recording, reproducing, or broadcasting of moving visual images.
Text-to-speech or speech synthesis is the artificial production of human speech and is generally accomplished with special software and/or hardware.
Assessment is the process of gathering information about a learner’s performance using a variety of methods and materials in order to determine learners’ knowledge, skills, and motivation for the purpose of making informed educational decisions.
Audio, in this context, is a digital form or representation of sound. It is a format that stores, copies, and produces sound according to the data in its file(s).
Multiple means of representation refers to the what of learning. Because learners vary in how they perceive and understand information, it is crucial to provide different ways of presenting content.
Multiple means of action and expression refers to the how of learning. Because learners vary in how they express their knowledge, it is crucial to allow them to express what they know in different ways.
Self-regulation is the ability to strategically modulate one’s emotional reactions or states in order to cope or engage with the environment more effectively.
Executive functioning refers to a set of cognitive capabilities, associated with the prefrontal cortex in the brain, that allow humans to overcome impulsive, short-term reactions to their environment and to instead set long-term goals, plan effective strategies for reaching those goals, monitor their progress, and modify strategies as needed.
