Rubrics: Students Like Them!

A recent paper on rubrics for online discussions had some interesting findings and in particular drew out some fantastic student responses about the use of the rubric for grading their online discussions, such as:

"I really appreciated having what was expected clearly documented at the beginning of the course. It let me know what was expected and helped me to prepare for each week. Other classes have seemed extremely arbitrary in their discussion grading and it can be very frustrating. I really enjoyed the way it was setup in this class. Thank you.”

The paper includes two versions of the rubric used and emphasizes several practices which both students and faculty found helpful in implementing and using the rubric:

http://jolt.merlot.org/vol7no4/linardopoulos_1211.htm

Posted by Clark Shah-Nelson at 4:05 PM - Categories: Distance Learning | Online Courses | Teaching Tips | VARIABLES before Assessment

B is for Big Ideas—Teach Them First

In many fields there are a lot of concepts with a lot of nuances.  Economics is no exception.  The basics of economics are fairly intuitive.  People generally respond to incentives.  When prices go down, people consumer more of something.  When prices go up, people consumer less of something.  When it comes to businesses, when the price of an input goes up, less is used.  And to get businesses to produce more of something the price must be higher.  We can talk about generalities and how the incentives are very straightforward and extend it to equilibrium and risk without too much effort. 

Then comes graphing and equations.  Those tend to trip people up a bit more.  People also tend to get tripped up by the need to consider more than one incentive at a time, although as long as we keep the conversation in terms of directions of change things work pretty well. 

So, how does this relate to the concept of big ideas.  The big idea is that incentives are important.  In a public health context, there may be plenty of things that public health professionals think that people should do.  The first through may be that simply making something more available will mean that people will use it.  What is forgotten in some cases is that people tend to do things when they make economic sense.  Or, when the incentives point in the right direction.  When we plan for how much of a service to make available, we should not base our planning on how much we think will e used because people should use something, but on how much we predict will be used because people respond to incentives that they have been given.  In a first health economics class (particularly when I have only eight weeks to teach students) if a student comes away with the idea that incentives matter when making decisions about medical care and public health and leaves behind the idea that people will just do something because it is recommended or right that is a  major accomplishment.

Once a student has learned that, they can then learn equations and graphing and nuances.  Learning these can take an entire career—or at least an entire course of PhD level graduate study.

The key for me is to structure learning opportunities—including lectures, readings, practice assignments with constructive feedback, and graded assignments to provide the opportunity to learn the big ideas first and then to learn the finer points.  This should permeate the planning process, the expectations that are set, the way that feedback is given, and the eventual assessment of the students.

Posted by Kevin Frick at 5:04 AM - Categories: VARIABLES before Assessment

A is for Application

Students in a school of public health usually want to know how what they are learning can be applied in the real world.  When I discuss an application of the material, I like to be able to point students to an example not just of the way in which the material MIGHT be used but to an example of the way that the material IS being used.  This is how we make things real.  Students are looking for a reality.

I recently received a comment in a course evaluation that demonstrated that at least one student had not been convinced of the applicability of the material.  Many other students have commented that they could not understand how someone would not realize the application, but even having one person not get it suggests that there is room for improvement.

In the course, students were asked to evaluate the quality of several cost-effectiveness articles.  The comment in the course evaluation was that the evaluation of quality criteria seemed like a rote exercise.  The next time I teach this, I will make sure to mention the following: in students' professional lives they may be asked (1) to review grants that include proposed cost-effectiveness analyses and will need to understand what makes  a high quality approach;  (2) to act as a peer reviewed for a manuscript submitted for publication, where they will also need to know what makes a high quality piece; (3) to participate as part of a research team to plan for a cost-effectiveness analysis so that they will need to understand what makes a high quality study; or (4) to use papers that have already been published to motivate policy change and they will need to asses the quality of articles that might drive quality change.  

I am considering asking students to offer a final thought at the end of each assignment in which they draw a conclusion about quality combining the multiple points they have assessed rather than simply assessing the quality criterion by criterion.  This would help to make students more aware of the need to go beyond simply reporting on individual criteria and to make the assignments feel like it is closer to a real world setting.

Posted by Kevin Frick at 10:19 AM - Categories: VARIABLES before Assessment

I is for Instructions

Students have a right to expect instructions that are clear and that are not changed during the course.  I know that I and my TA's have tried many times to write questions for my health economics class that can be interpreted only one way.  Try as we might, it turns out that they can often be interpreted more than one way.  When students are working through the problem and not facing a multiple choice test, they have no problem telling me about the alternative interpretation of my instructions.  The students with the alternative interpretation, their classmates, and the instructors all find this difficult and time consuming to deal with.  This is just on illustration of the need for and value of clear instructions.  

Students like to know due dates.  They rarely complain when the due dates are moved to a later date.  I don't think I have ever shifted a due date to an earlier date.  Still, students in the setting in which I teach are taking classes in eight week segments during which they are often facing 20 hours a week of contact with faculty.  This is a huge time investment and needs to be planned for.  Thus, making sure that the instructions are as clear as possible on day one of a course is particularly helpful in my case, but this is certainly generalizable.

I think that the giving of instructions can also be used to set expectations.  A constant struggle at this point in time is students in class with laptops.  I know that many students like to take notes directly on their laptops.  Some schools have even invested in software that facilitates direct annotation on pdf files.  However, some students when given the choice between paying attention or using the wireless Internet will choose to use the wireless internet for non-class-related purposes.  On the one hand, students paying a lot of money for their education have the choice of how to spend their class time.  On the other hand, and here it is particularly useful to teach economics, there is an externality that is associated with non-class-related use of the Internet.  It does not affect only the student who is using the Internet but the student's classmates as well.  This would be a great way to introduce the concept of externalities on day one of class (which is also particularly relevant in public health) and then to set the expectation that students not do this.

What other expectations should be set?  I like to set expectations about participation and taking advantage of the learning opportunities that are presented.  I am a long distance runner who is now training for my third half marathon with an eye toward my second full marathon this October.  In the 13-20 week training programs for such events there are plans for running and fitness opportunities for every day. Including, on some days, resting.  If the runners follow the directions of the coach, it maximizes the probability of success for the runners.  If we think of the students as analogous to the runners and the instructor as analogous to the coach, the instructor plans a series of learning opportunities.  The students need to be encouraged to do them all--to take advantage of them all--so that they will understand the bigger picture in which the set of activities fits.  Then, the students will maximize their chances of learning and of deep understanding.

In economics and economic evaluation, I particularly focus on giving students instructions on how to use the practice examples that they are given, questioning and discussing the premises of economics and the applications of economics, and using opportunities for group learning appropriately to help with understanding the material.  Just a few comments on those.  I try to give students two practice examples.  One that they will most likely do on their own.  A second in a group. Then they get assessed.  I'll discuss this more later.  Some students don't find the time to do the practice.  If they don't they will miss an important learning opportunity.

I don't teach economics as dogma.  I want students to question it and I want them to question its application.  Putting it to the test is the only way to really understand it.

Finally, I encourage students to work together but complete assignments on their own.  There is a time and a place for group learning as long as the final product is each student's.  Again, setting clear expectations about the appropriate use of group learning opportunities is a great thing to include in the instructions I give to the class each time.

There are many creative ways in which instructions can shape the entire learning experience rather than simply being a drag to read through a syllabus on day one or rather than foregoing the opportunity to give instructions and jumping right into the class.

Posted by Kevin Frick at 8:46 AM - Categories: VARIABLES before Assessment

R is for Realities

When I begin a new class, I find it useful to understand students’ realities.  In other words, to understand how they might use the information from the class in their professional lives.  If I were teaching in an undergraduate setting, it would not necessarily be clear how students might use the information in their professional lives because very few would be certain of what their professional lives would be.  In contrast, in a graduate setting in which many students view life through a very applied lens, like a school of public health or most of the entire medical campus, many students are returning professionals or otherwise have a pretty good idea of what their professional lives will be. Students rightfully want to know how what I am teaching them can be directly applied to what they are already doing or are going to be doing in their jobs in public health and medical care.  Gathering information about students’ motivations for learning in general was part of understanding general variation that I discussed in an earlier entry.  Gathering information about students’ motivations for a specific class is part of this step on realities.  If a student has a direct application of the material to their professional activities, then, even if the student is generally looking to pass tests and earn a credential, the student may be more willing to engage.  In contrast, if a student is generally a person who loves to learn but sees no way in which he or she would ever use the information from a particular class, the student would be less engaged.

Understanding student realities can also facilitate providing examples.  I don’t use only nursing examples when I am addressing nurses in a classroom, but I am certainly more inclined to do so.  I don’t use only health administration examples in classes in which I have a lot of students who are earning Masters of Health Administration degrees, but I am more inclined to discuss a hospital or health insurance example.  Similarly, when I have a large group of MPH students, I’ll make sure to include examples of public health economics or economic evaluations in public health settings. 

Beyond professional activity differences, what are other areas of realities that are of concern to me?  Some students are not from the United States.  While they may find applications from the United States health care system to be interesting, they are also interested in knowing things that are specific to their own countries or at least to countries that are more like theirs.  Even within the United States the reality has changed greatly over time.  Lecturing about a chapter in a text book that describes an out of date reality may be an interesting historical use of economics but does not provide students with direct insight about the world today. 

In addition to planning for the course in the first place having an awareness of students’ realities allows me to better plan for questions that may be asked.

Finally, there is the question of how to apply this information.  One example came from a health economics class.  I shared with students some information on how a particular health care system had responded to a shortage of some pharmaceuticals for pain control and anesthesia.  The article seemed rather concerning, so I thought that an interesting reality application would be to use economics to interpret the events in a letter to the editor.  I gave students the opportunity to structure their letter in ways that would reflect their reality.  The students were allowed to choose from a number of different perspectives (e.g. physician leadership, patient advocacy, or hospital administration) and then were asked to discuss the most relevant economic points.  I’m not sure whether each student chose the perspective closest to his or her own reality or tried a bit of role playing in their response, but it gave the students an opportunity to shape the assignment to their own realities and that is the best I think I could do in that situation.

Posted by Kevin Frick at 5:32 AM - Categories: VARIABLES before Assessment

A is for Awareness

The second step in preparing to teach is Awareness.  As a practical matter, this means:             

·     understanding the students' priors about the material in a course. How well prepared are they?  Do they have misconceptions?

 So, the first question is whether there is a prerequisite for the course?  If there is, how is the prerequisite structured?  Is it structured to get students to memorize a lot of facts so that they should be familiar with the vocabulary of a course but may not understand the concepts deeply enough to use them in new and creative ways?  What is the variation in the grades in the prerequisite?  Have students practiced using the concepts before? Have they worked with the concepts at the level of explaining them and writing about them?  Have they just done problems but not had to explain their work?  There are so many things that will vary about students "priors" with respect to the course materials.  A pre-test could be given, although this is not something that I am a fan of doing.

However, the pre-test concept does have the value of indicating whether the students have an existing mastery and may help to turn up misconceptions.  There are other ways to get at misconceptions.  In the process of having students introduce themselves, they could be asked about impressions.  

When I work with students in the Doctor of Nursing Practice program, I like to ask them what they thought of economics before I have the read a book called The Economic Naturalist by Robert Frank.   The students in such a program are bright students.  They usually have day jobs in which they are managing departments or otherwise in leadership positions that require a lot of thinking.  However, they consistently indicate that their priors are that economics is highly mathematical, or very abstract, or has lots of jargon.  Few have ever taken more than a single economics course.  They think largely about macroeconomics and all sorts of confusing things that they hear about monetary and fiscal policy on TV or on the radio when it comes to economics.  They do not think about economics as a basic motivation for decisions in our lives.  They don't think about basic incentives that firms face every day.  When they have read the book by Frank, they comment that economics can be creative, or that they find it down to earth, or that the concepts really aren't all that hard, or that it actually makes a lot of sense.  This process of re-orienting their awareness prior to trying to lecture them about health economics is key to my working with them over the next several weeks.  It is also rather painless and opens them to the learning experience.

I have another great example of my "Aha!" moment when it comes to awareness and priors and nurses.  I was speaking at a local Sigma Theta Tau research conference in 2005.  ??? is the nursing honor society. On that particular day in 2005, I was the second of two plenary speakers in a row.  I listened carefully to the speaker before me and found a way to link what she was saying to my presentation.  I spent about 5 minutes at the start of my presentation making this linkage.  I had not planned to do that but it worked.  As I then went on to describe economic evaluation, the attendees facial expressions and body language suggested that they were understanding me in ways that other audiences (of nurses or otherwise) had not when I had tried to make similar presentations before.  That was the moment at which I realized just how important starting from what the audience knew and building a bridge was so important.  It was ironic as I had been working with nurse researchers for six years by that point, but every time I tried to explain what I did I used lots of graphs and calculus and tried to explain things but never really tried to build a bridge piece by piece.  My approach had been more like throwing a rope wildly that the nurses on the other side of the knowledge gap were supposed to catch and hang on to rather than building a solid bridge from what they knew already to what I wanted them to know.   

Looking at my own teaching, I may be taking for granted that students have a higher level of understanding of microeconomics when I try to teach health economics.  I may need to go back and reassess how I am teaching so that I can help to solidify the basic microeconomic concepts.  Then, I could put my own approach into practice in all the courses I teach and not just in some of them.  

Posted by Kevin Frick at 9:30 PM - Categories: VARIABLES before Assessment

V is for Variation

The first term in VARIABLES before Assessment is Variation.  In this case, the specific thought process is to appreciate the variation in the students' backgrounds when they come to a learning experience.  Failure to appreciate the variation and heterogeneity may lead to a great learning experience for some students—those who happen to be best attuned to the teaching style you are using—but is unlikely to result in an optimal learning experience for all students or for all students or for the class as a whole.  Sometimes it is impossible to know the variation prior to the start of the class and there are not always good ways to assess the variation even at the start of the class but attempts can be made.  Additionally, it is possible to make sure in advance that the learning opportunities that are provided appeal to students who learn in different ways, for example using different options among the “multiple intelligences” (suggested by Gardner).  In online classes, each student could be asked to simply provide a short paragraph about himself or herself that included key aspects about their learning process. 

What types of variation are of most interest: students vary in terms of (1) personality which may be associated with learning style; (2) general experiences and learning experiences; and (3) motivations.  Groups of students vary in terms of size and overall interactivity or “the personality of the class” based on how they interact with each other and with the instructor.

There is work using the Myers-Briggs personality descriptions that demonstrate that different students are likely to learn in different ways.  Some appreciate discussion and trying to imagine how to apply what they are learning and wanting to take things a step beyond what they read or have told to them.  Other students are mostly interested in facts and memorization and would prefer that instructors stay on a relatively straight and narrow path when they teach.  Some students do not particularly like to engage in conversation and find anything that is even a slight digression to not fit with their vision of the class or their expectations.  There are other ways of describing personalities and there are other ways in which personality might be associated with learning style.  Multiple intelligences and the idea that there are different characteristics of an individual and different aspects of how they learn was alluded to earlier.  Regardless, the key is to comprehend just how important this is.

Students vary in their backgrounds.  One thing that is likely to shape their preference for a learning style is the experiences they have had in the past.  These could be general experiences or experiences primarily in a learning environment.  The quality of students’ experiences in the classroom previously will greatly influence what they expect from an instructor and how they expect to learn.  Students will most likely hope for whatever is similar to their best learning experience.  Of course, we would hope that students are sufficiently adaptable to try new experiences and new learning styles, but we all have a comfort zone with activities in our lives.  So, an instructor should not be surprised when it is suggested that he or she should try something similar to a style of instruction and providing learning opportunities similar to what students have felt has been successful for them in the past.

Students also vary in terms of motivation.  Some really want to learn for the sake of learning and for the sake of understanding.  I am tempted to describe it as learning for the sake of knowledge, but it really is more than just knowledge that I am hoping to important.  It is understanding—how to use this again in the future.  Other students are not nearly so worried about the lifelong learning process but are focused on learning what they need to know to pass a test.  The test is just a step on the way to a grade.  The grade is just a step on the way to a credential.  The credential is the focus of their effort rather than the focus being the learning itself.

Groups of students also vary.  And each student may act somewhat differently depending on the nature of the group. The size of the group is at least a partial determinant of the level of interaction that can be expected—particularly from each student.  In a class of 100, it is impossible for each student to say as much as he or she might in a class of 10.  The instructor will be much more likely to have to cut off a conversation and just move on in a class of 100.  I think about this a lot as I try to encourage interaction but realize that it must have its limits in large classroom settings.

Finally, each class takes on its own personality.  The number of students, the personalities of students, and the ways in which a small number of students are the dominant ones asking questions all have an influence on how the learning process will work for the group.  It is critical for me to get a feel for this as I move forward in teaching a class.  This is definitely something that can only be ascertained after several lectures, but it is critical to try to shape the personality of the class and to guide the interaction not just with each student but with the class as a whole.  Conversation should be productive and not antagonistic.  Conversation should focus on supplementing and extending the discussion.  Conversation should not be forced.  All of these are critical considerations.

Posted by Kevin Frick at 5:14 AM - Categories: Teaching Tips | VARIABLES before Assessment