Facultyware Choose Page UDI Information (will open a new window)



UDI and Usability: Preparing to Review Instructional Products

 In the development of a juried review process for the Universal Design for Instruction project, it became apparent that two factors needed to be considered. First, it was important to measure the extent to which a submitted product displayed each of the nine Principles of UDI ©. However, given that products will be available for use by postsecondary faculty from a range of institutional types and subject disciplines, it was also important to measure the quality and usability of each product. To achieve these multiple goals, two rating forms were developed. The first measured the extent to which a product reflected each of the nine Principles of UDI ©, and the second measured the quality and usability of a product. These rating forms were developed initially in text format, and were distributed to 20 professionals in postsecondary education across the United States. Feedback was obtained from 11 of these professionals, and both forms were revised accordingly.



<scroll for more>
Return to the newsletter home page
UDI Updates
Facultyware Choose Page

As these rating forms were in development, project staff began identifying reviewers. It was decided that two panels would be established, each of which would handle a different portion of the review process. The first panel convened was a group of national experts in Universal Design. This panel was tasked with rating the extent to which a product reflected the nine Principles of UDI ©. For the second panel, project staff decided to tap into the expertise of the UDI team members at project collaborating schools. These team members were assigned the rating of product quality and usability.

An exciting factor that emerged during this planning process was the availability of the Facultyware site to serve as a vehicle to present products to the review panels, and to receive immediate on-line ratings. The two rating forms were placed on-line via Microsoft FrontPage. Password secure areas were established for both panels of reviewers, to ensure that only individuals who participated in the UDI orientation process were reviewing products and sending ratings to the project team. Detailed orientation modules were developed for each review panel. These modules were available in both paper and electronic format, and provided instructions about accessing the secure area of the Facultyware site, how to view posted products, and how to access and complete the appropriate rating form.

With the two expert review panels in place, the logistics of the review process were established as follows. Project staff initially review submitted products. This is a broad review, designed to ensure that the product is designed for postsecondary education, and that sufficient information about the product and the submitter are provided. If the product passes this stage of the review, it is posted to the secure area of Facultyware. The UD expert panel then conducts an on-line review; again, according to how well the product reflects each of the nine Principles of UDI ©. Because many products may be reflective of some, but not all of the principles, the product is continued to the next stage of review if the mean rating is 3.0 or higher (out of 5.0). If the product passes this stage of the review, the experts on the UDI teams nationwide review the product on-line. To pass this second review stage, the mean rating of the product must be 3.0 or higher (out of 5.0). If the product passes this final review stage, the product and its rating scores are posted in an open area of the Instructional Freeware section of the Facultyware site. This allows the product to be accessed by anyone who visits the Facultyware site.

In March and April of 2001, a pilot testing of the review process was conducted. Both review panels rated three instructional products; two text documents and one audio interview with a university professor. The product ratings were examined not only for the mean scores in each designated area, but also for broad themes and trends in responses. This analysis helped project staff identify areas of concern among product reviewers. For example, it became apparent that reviewers would benefit from additional background information on the course from which the product was derived. Furthermore, feedback was sought from members of both review panels to ensure that the training modules and rating forms were clear, and that the accessibility of the electronic rating process was acceptable. Relevant modifications were made according to this analysis and feedback.

Results of the pilot process can be found in the “Instructional Freeware” section of the Facultyware site. Results are presented on two levels. First, a summary of key findings is presented, including comments related to product strengths and areas in which the product can be modified to be more reflective of UDI. A hot link is then available for detailed ratings and reviewer comments. Additionally, links are available to key information, such as the definition of a “product,” both of the product rating forms, and the orientation modules for both the UDI expert panel and the faculty review panel.

This electronic review process will provide the UDI project staff and product submitters with a rapid turnaround on product review. The true beneficiaries of the review process, however, will be visitors to the Facultyware site, who will be offered instructional products that are of high quality and that exemplify the Principles of UDI ©.