The Assessment Plan: Creating Measures to Collect Data/Evidence
Assessment activities or measures should align with learning outcomes and clearly answer the question: How can we check that this learning outcome is being met by our students? Start by defining one direct and one indirect measure for each outcome. You do NOT need a separate assessment for each outcome; some assessments may measure more than one learning outcome.
Data-collection methods for assessment purposes typically fall into two categories: direct and indirect measures. Direct measures of student learning come in the form of a student product or performance that can be evaluated by a knowledgeable instructor or practitioner. Indirect measures are the perceptions, opinions, or attitudes of students (or others). Both are important, but indirect evidence by itself is insufficient. Best practice is for a program to collect both types.
- Data-Collection Methods: Direct and Indirect
- Benefits and Drawbacks of Data-collection Methods
- Evaluate Your Choice of Method and Targets
1. Data-Collection Methods: Direct and Indirect
After faculty members agree on the student learning outcomes for a program, they need to choose data-collection methods that will provide evidence of learning that can aid in decision making.
Why is direct evidence of student learning required? Here's an example. If students self-report on a survey (indirect evidence of learning) that their knowledge of world geography is excellent but later fail a multiple-choice world geography test (direct evidence), that's useful information. The indirect evidence by itself is not as meaningful without the direct evidence of students' knowledge. Direct evidence, by itself, can reveal what students have learned and to what degree, but it does not provide information as to why the student learned or did not learn. The why is valuable because it can guide faculty members in how to interpret results and make improvements. Indirect evidence can be used to answer why questions. Programs should collect both direct and indirect evidence of student learning to gain a better picture of their students.
- Choose methods that will
- answer specific assessment questions,
- be seen as credible to the faculty and the intended users of the results, and
- provide useful information. Quantity is not the goal.
- Use more than one method whenever possible, especially when answering questions about highly-valued learning outcomes.
- Use or modify existing learning tasks; e.g., assignments, projects, or exams, whenever possible. Inventory what evidence of student learning and perceptions about the program already exist. The curriculum map is a useful tool when conducting the inventory.
- Choose methods that are feasible given your program's resources, money, and the amount of time faculty are willing to devote to assessment activities.
2. Benefits and Drawbacks of Data-collection Methods
When selecting the best method(s) to answer your assessment questions, take the benefits and drawbacks into consideration. Think about the following:
- the method's consequences (intended and unintended),
- whether the method will be seen as credible by the faculty and intended users of the results, and
- whether faculty and users will be willing to make program changes based on the evidence the method provides.
Some methods have beneficial consequences unrelated to the results of the evaluation. For example:
- Portfolios: Keeping a portfolio can lead students to become more reflective and increase their motivation to learn.
- Embedded assignments: When faculty members collaborate to create scoring rubrics and reach consensus on what is acceptable and exemplary student work, students receive more consistent grading and feedback from instructors in the program.
3. Evaluate Your Choice of Data-collection Method and Targets
After selecting a data-collection method, use this checklist to help confirm your decision. A well-chosen method:
- Provides specific answers to the assessment question being investigated.
- Is feasible to carry out given program resources and amount of time faculty members are willing to invest in assessment activities.
- Has a maximum of positive effects and minimum of negative ones. The method should give faculty members, other stakeholders, and students the right messages about what is important to learn and teach.
- Provides useful, meaningful information that can be used as a basis for decision-making.
- Provides results that faculty members and other stakeholders will believe are credible.
- Provides results that are actionable. Faculty members will be willing to discuss and make changes to the program (as needed) based on the results.
- Takes advantage of existing products (e.g., exams or surveys the faculty/program already uses) whenever possible.
The final step in developing your measures is to decide what your acceptable and ideal targets will be. A simple way to determine the targets is to ask your faculty the following questions:
- What is the ideal level of achievement you want your students to meet to consider your program successful?
- 100% achieve the highest score possible?
- 90% achieve the average score
- If a pre-test, maybe 50% of the students will score a 60% or better, is that ideal?
- What is the acceptable level of achievement you want your students to meet to consider your program successful?
- 80% of the students score at average or higher?
- 75% of the students meet 3 of the 4 rubric criteria?
These are just a few examples of questions you can ask yourselves. Basically, you want to set the levels at which you are comfortable for your program. If students drop below the acceptable target in a measure, then you can consider using the findings to make changes to improve student performance.
Adapted from the Assessment pages of the University of Hawaii, Manoa.
If you have any questions about assessment of student learning, please contact the OIA Assessment Team.