Session 9: Reviews and Feedback

Here are the links we used in our meeting 4/14/2022 meeting:

The ultimate purpose of review is to ensure that your OER is well-structured and ready to be used in the classroom. Review can help you get critical input and suggestions for change that will make your OER even stronger. By sharing your book with subject experts, you can ensure that the content is appropriate, accurate, and adequately covers the material. At heart, review is about bringing more hands on deck to invest and help your resource.

Today’s session covered the different kinds of review, workflows for these processes, and important considerations for this stage of your projects. We looked at a few central documents and questions that may support you all, and also do a bit of forward thinking about how to share the results of this process!

  1. Peer Review: ‘Peers’ can offer constructive feedback and solutions to improve the quality of educational content. We encourage you to reflect, recognize, and minimise biases in peer review. For instance, consider what types of feedback you need and who can speak to the quality of your content besides another instructor — would an industry expert be able to input? Think back to your SLOs — whose subject matter perspectives are needed to help determine whether the OER is built to help students achieve these outcomes?
  2. Accessibility Review: The accessibility review involves a thorough run through the different output formats of your OER looking specifically at the web accessibility in each format. A specific set of accessibility criteria can guide the people in your team who are tasked with this form of review to ensure that your resource meets the desired accessibility standards. The goal is to make as accessible an OER as you can, knowing that there is always opportunity for improvement down the road.
  3. Classroom Review: This form of review is particularly powerful because it invites feedback from the students which ultimately will help your team to determine necessary improvements for future iterations. Feedback can be gathered both from the instructor using the book to teach as well as the students using the book to learn. Try to identify some academic and non-academic measures as you gather comments from the classroom.

We provided a Review Guide Template that will help you establish review workflows and identify expectations and central guiding questions to better structure your review process and support reviewers. There can be many different lenses/criteria to keep in mind when reviewing the resource, and we suggested coming up with 3-5 central questions to keep things manageable. This is laid out in more detail in the handout for session 9.

In the final part of our session, we asked you to think ahead to how storytelling can be used to communicate the quality of your resource. We prompted you with discussion questions to help you and your teams think of ways to center equity during the review process. We want to compile all your answers here so you and other cohort participants, in this group and in concurrent cohorts can share and learn from one another. Thank you for sharing your responses as replies to this thread below.

While this stage is fairly straightforward, it’s critical to prepare all the documents and workflows ahead of time to ensure smooth sailing. And remember: along the way, if you have any questions - do not hesitate to lean on each other and the open community, including the Rebus forum, cohort members, and myself.

Next week, we’ll begin looking towards the book’s official launch with a session on formatting and release preparation. This phase is one where your project really begins to take shape as a whole, usable resource.

Questions we used for our discussion activity:

  1. How will your team attempt to manage the effects of bias in the review process?
  2. How will your team work to invite a more diverse range of reviews and value a broad range of perspectives?
  3. Are there non-traditional subject matter experts you’d like to work with? (students, community members, etc.)
  4. Are there project specific questions you would like to ask during the review process?

Chat transcript

  1. We will be conscious of who we approach for reviews, making sure to have a diverse group of people with various expertise and backgrounds and levels.
  2. I think the three of us will need to put our resources and connections together.
  3. We will rely heavily on student reviewers as we are most concerned with whether the content is accessible, legible, engaging, and also issues with tone.
  4. Since our book is a handbook focusing on technical instruction, our main question would be toward people who are not familiar with photography: Does the material help you understand the principles and do the processes without error?

How will your team attempt to manage the effects of bias in the review process?

I am going to use the APA bias-free language guide for sure. And I will rethink some of my cases for greater diversity, though rice is the centerpiece, which is a foodstuff just about every culture partakes of.

How will your team work to invite a more diverse range of reviews and value a broad range of perspectives?

+ I will seek diversity in reviewers, both in students and faculty and staff. CCNY is excellent on that front, for its universal representation.

Are there non-traditional subject matter experts you’d like to work with? (students, community members, etc.)

+ I am hoping my medical school (history minor) student will be able to help with the science research … But non-academic, I have to give it some thought. I hadn’t otherwise considered the non-traditional contributor. Thanks for that.

Are there project specific questions you would like to ask during the review process?

+ I am not sure I can answer that at this point. I have to see what the product looks like as we move along.

  1. As support staff, I’ll help the teams I’m working with figure out what perspectives might be missing from their reviews. I can also help them find reviewers, and make connections!

How will your team attempt to manage the effects of bias in the review process? We’ll be considering DEI guidelines. By its own essence, a World Music course is compliant with DEI. It main objective is to create multicultural awareness, and by extension to deal not only with “multicultural” issues (a term a bit antiquated, but any issue pertaining to intersectional identities (class, gender, etc.)

How will your team work to invite a more diverse range of reviews and value a broad range of perspectives? I think classroom use of the OER is fundamental. In addition to expert, peer review, we can learn a lot by using the book in a real classroom and see how students respond.

How will your team attempt to manage the effects of bias in the review process?
We will work to invite reviewers that represent a diverse range of identities, disciplines, and perspectives.

Are there non-traditional subject matter experts you’d like to work with? (students, community members, etc.)
Yes! I am especially interested in working with graduate and undergraduate students in NYC.

Are there project specific questions you would like to ask during the review process?
We are particularly interested in how the different languages, dialects, and codemeshing we hope to include in the book will be perceived by readers. The last thing we want to do is fetishize folks’ different languages. And I know we simply won’t be able to represent all voices. Plus, I’m painfully aware of how much “vernaculars” are perceived as unqualified, unprofessional, and unfit for academic settings, and textbooks are particularly set on using so-called “standard” English. I suspect that this is going to a sticky issue needing our careful attention as well as reviewers’ thoughtful feedback. I think this invites us to also diversify our reviewers: Reviewers versed in my field may be ready for such linguisitic diversity in a textbook, while new teachers of composition who are unfamiliar with this research may be skeptical. Then, students themselves will be key reviewers since I’ll be very interested in how they experienced the different languaging we’ll include. In my teaching, I have found some students to be, on the one hand, open to linguistic diversity yet, on the other hand, still uncomfortable seeing it written in print. This will be interesting.

Thanks, Julia – good to know you have resources for reviewers … I know I will eventually need a scientist to review the nutrition science literature.

1 Like
  1. How will your team attempt to manage the effects of bias in the review process?
    One can address bias by making cvlear review guidelines available to reviewers.
  2. How will your team work to invite a more diverse range of reviews and value a broad range of perspectives?
    One way to think about diversity of review is to broadcast to international colleagues who will have different perspectives. Another way is to consider what diversity means for the project. So inviting colleagues who champion different hypothesis in the field could be useful, although subject to controversy. I am more inclined to invite reviewers from different fields like plant physiology, animal physiology, fungi physiology etc…
  3. Are there non-traditional subject matter experts you’d like to work with? (students, community members, etc.)
    Potentially graduate students, postdocs. By definition one could argue that undergraduate students are not experts, but could consider undergrads from certain backgrounds, or from academic programs such as honors students and Biology club students. This could also potentially introduce a bias.
  4. Are there project specific questions you would like to ask during the review process?
    Yes. this would ask reviewers to provide conceptual feedback.

I can definitely help with that!