Here are the topics of this seminar announcement, a combination of my notes (and additions) and those of Christy's from our last meeting Nov 15.

I. RETENTION vs ASSESSMENT:
I.a. Identifying if the problem we would like to solve is retention:
I.b. Identifying what ideas we bring to assessment that makes it a solution:
II. OVERALL TECHNOLOGY TOOL:
II.a. Response to #1: More information on IQs and WWs:
II.b. Response to #2: Grading by the Instructor and Feedback to the Student:
II.c. Response to #3: Training others in the use of WWs and IQs:
II.d. Response to #4: Reporting on Results:
III. EDUCATIONAL AND CURRICULAR TOOLS:
IV. GATHERING STUDENT PROFILES:
V. MORE ON THE POTENTIAL FUNDING SOURCE:

I. RETENTION vs ASSESSMENT:

I.a. Identifying if the problem we would like to solve is retention: 

If the problem is retention – surely the 52% statistic on MSU-B dropout from the 1st to 2nd year is staggering – then we need some analysis of this. Here are some analytical questions:

I.b. Identifying what ideas we bring to assessment that makes it a solution:

The assessment tools as part of the solution posed by retention. Here are some points about how we would use the assessment tools we have discussed. To simplify I discuss at first the use mainly of the two Assessment tools W(eb)W(ork)s and I(nteractive) Q(uestionnaire)s.

The design of IQs is to question in steps, and to offer hints about how to go between the steps. It is the toughest part of learning in math/science/engineering. If you don't offer a scaffolding of support to step-thinking, no students will pick up the idea. Rather, almost all will memorize every possible test situation whole cloth. The result: Such memorization is an overload, and students remember nothing of the analysis that makes it possible for experts (the instructor) to reconstruct the situation based on reusable reminders. By reusable reminders, I mean often the basic – usually small number – of fundamental principles on which a course is traditionally based.

II. OVERALL TECHNOLOGY TOOL:

Christy and Mark said they needed more info on WWs and IQs. Presentations I have already given on WWs and IQs can be given again – to them – in the Education Lab, on their overhead projector.  See III for more on that. After such a presentation the likely questions would include the following:
  1. How do we set WWs and IQs up to run on campus?
  2. How do we enroll instructors into their use?
  3. How much administration would they require?
  4. How do we gather data from the results of their use?

II.a. Response to #1: More information on IQs and WWs:

In response to #1, WWs would be put on a "server" (preferably a Mac) and run through the Network administered by Mike Barber. With any failure on that, it could also be run through the server at UCI managed by Domingos Begalli, with the understanding they would be part of the grant: We would have to use resources here to handle it. Further discussion would be on what resources are necessary.

I explain in the following paper how IQs work, including the style of questions, and their use through e-mail:

On my home page, section: II. Education Assessment Work and Vita
      --> *  Education articles:   U(nix)O(ffice)S(ystem), E-mail Technology, I(nteractive)Q(uestionnaire)s
      --> Item #2
Interactive E-Mail Assessment, MAA Vol. on Assessment, B. Gold, S.Z. Keith, and W.A. Marion, eds., Assessment in Undergraduate Mathematics, MAA Notes #49, Wash. DC, 1999, 80--84.

Here is how a student saw them.

Students could save their answers temporarily in their own accounts. So they could do multiple attempts to answer an IQ. The IQ program did this transparently, so the students only needed to start up the IQ again to see their answers.

When they were done they indicated that, and their personal IQ response was then mailed to me. I then added their IQ to their personal Portfolio of Interaction. Those Portfolios of Interaction were the main tool for my finding out the main difficulties with courses, and the success of using IQs to intervene in the usual – never look back – march of a curriculum to the end of the book. The paper above discusses this.

II.b. Response to #2: Grading by the Instructor and Feedback to the Student:

In response to #2, the presentation I alluded to above shows graphically and convincingly to most who have seen it, that the experience of growth of use at UCI of WWs was phenomenal.

One reason: It dropped the amount of instructor grading while increasing by an order of magnitude students satisfaction with the amount of feedback and adequacy of the assessments on which they got grades.

Since IQs test for a tougher type of thinking, reading the results to those uninitiated in the IQ rubric must at first think that it is like going back to hand-grading. It is not for several reasons. First: The analysis revolves around going between the structured steps, so it is possible to use within each step some of the same grading techniques used in WWs. Yes, there is actual writing that takes place – usually in the last step of IQ questions. Such steps are there to entice students to see they have been going through a process to answer the question, and little of their answers entailed rote memorization. (Help aids, for example, can be included. A student who uses them can be docked points on that question, but still they can go on. Some help aid remind them of relevant definitions for free.) Those final steps often alert the instructor to the few who catch a lot of what are the main issues. That is information for the instructor.

Conclusion: Use of IQs is for those who are trying to get students to see the relation between the pieces of their course, and even between courses. I claim – as happened at UCI – that students trained on IQs will see it possible to go to the next course in a series. IQs are a retention-in-a-discipline tool.

Second: The last piece of each question asks students to write analytically about what they learned from the previous parts to answer a specific question. What some students get from this: They could answer the specific question – sort of on their own, even though, at they start, they were clueles what the answer would be.

But, you ask, "Don't you have to grade such writing responses in the old fashioned slow way?" See the end of Part II.c.

II.c. Response to #3: Training others in the use of WWs and IQs and Reporting on their outcomes:

In response to #3, I've already made the strongest case I possibly can that some form of help desk is necessary. People who are new to technology make strange beginning mistakes. Some things happen that cause them to have no confidence. I give a story about when I first introduced IQs, and sent them around to 10 faculty members who knew what I was doing because we'd had lunch over it, or they had been especially responsive at a campus talk I had given, or in some cases they were just good, interested friends.

So each of the ten received an IQ and were supposed to launch it. Upon its conclusion it would return to me, and then I would know it worked.

After a week I had received only one response, from Steve Franklin – immediately. Steve was a techie from IT. The next week I called up the other nine from my office. Each said it didn't work. I arranged meeting with them in their offices, one group of two, another of three, and the 3rd with four (they were conveniently in the same building), to see what was the problem.

In each group I had one from the group sit at the computer, and type in the command that would launch the IQ (from the Unix command line in their account). In the first group, after the person typed, pressed return and it didn't work, I suggested they look carefully and type it again. They did, mispelling the command name wrong for the second time. I showed them how to set their unix command line so they could recall a previously typed line.

Then, we looked at their spelling which I then changed. Upon pressing return the IQ launched as predicted.

In the second group the mispelling occurred when my proxy for the group named the mail file. It took no longer to correct, though it was a slightly different error. The 3rd group was the punchline. The person I chose for my proxy was Larry Crystal. I had hired Larry to administer testing ideas from my grant. He was a high school teacher who had run summer programs for K-12 teachers with me for three summers.

He typed what the instructions had told him, and it worked perfectly. Larry, however, said: "See it doesn't work!" I said, "Larry, it's working just as I told you it would!" He was puzzled, then looked again, and saw it was working. His response: "Well, of course it worked for you. You knew it would!"

Larry has been the manager of WWs at UCI since that time. He also handles all the WW courses in mathematics. There are conclusions to draw from this example, but I move on to IQs.

IQs are more ambitious and that is because they have two innovations that require training beyond just learning to launch them.

First: Making them is a different mind set. I told a story in our last meeting about using IQ type questions – not IQs themselves – in my department to construct admission-to-PhD-candidacy exams in the core disciplines. I won't retell that here, except to remind of the punchline. 

Not only did they gave an astounding improvement on the pass rate, but my colleagues predicted the result would be far worse than the previous abysmal pass rate. Why were they so wrong? Answer: Because they didn't understand that students could pass without having fully memorized answers. How could that be? Answer: I will answer that at another time.

Finally, IQs require some training in using an IQ-grader. What the IQ-grader (a different grader for each IQ question set) does is allow batch grading. You can grade written answers in even large classes efficiently by having the grader put clusters of answers from many students up in front of you at one time. My article calls this ripping the blue-book apart. Here is what you find.

Not only is it easier to read – what an understatement that is – typing over handwriting, but when you see, say, 15 answers to the same question at one time you often see the same mistake repeated, maybe 10 times.

You then do the following:

What the student gets: A full answer to what was wrong with their question, typed by you once, but repeated 10 times. Students don't traditionally get full answers written on their tests to involved questions, but you knew that didn't you?

Then, you tell the IQ-grader to drop those answers off the page, so you can now concentrate on the remaining five, or take on another batch of 15, etc.

There is another even more effective procedure for batch grading. It gives the instructor much less work, and manages in the end to make more of a carefully written IQ than the above. I'll leave that for another time.

Conclusion: I hope my examples allow you to conclude that serious technology requires some training. So you must be asking where is this help desk set up? Answer: Of course it is done with e-mail, and it would be part of our grant to have the ability to report on the success of the training.

Though it is not as sexy as IQs, I believe you would be astounded at how effective e-mail programs can be to handle interacting with large groups, or small groups often on many issues. That I also offer as a presentation.

II.d. Response to #4: Reporting on Results:

The results – student answers – from WWs and IQs are text files containing tagged data, that can be manipulated into reports. As with all the reports I mention, you expect to work hard on getting these educational experiments to work. You hope to have reports come from it fairly automatically.

Like the e-mail programs this is run by a set of programs that use html-looking tags to allow automatic collation of parts of the work into reports. An example of using the tagging system is illustrated in Matt Peterson's graphic, called Polling the Portfolio. The idea is that the IQ-grader for a set of problems can go back into the student portfolios to put together a report on the outcome to particular questions.

You especially want this on the questions that are causing classroom bottlenecks, whereby you find – late in the quarter – it seems no one remembers the topic. That is the time to intercede with the class, and remind them of their previous work with that topic. Resonant topics blow classes away. It always astounds instructores, who test for it, how often you thought you had covered something well, and no one recalls it at all. My graphic Dynamic Learning Curves on a Difficult Topic illustrate the difference in using portfolio polling and then another IQ intervention, to handle this. This is the big impact area on traditionally difficult classes.

III. EDUCATIONAL AND CURRICULAR TOOLS:

I've alluded to two presentations I might give you above. These presentations could be considered as part of our meetings – excuse me, our seminar. Though we are a small group, seminars on most campuses are small groups run by energetic people, with an occasional highlight event.

The presentations I have in mind are to show the value of running everything under UNIX. This is easy to do on a mac or on a PC. To see an HTML abstract on such a presentation follow this path.

On my home page, section: II. Education Assessment Work and Vita
      --> * Technology course: Class on using Assessment and E-Mail Technology from my Education Articles
      --> Item #1
Talk titled, Connecting the Dots for a Quality Education given in the Beartooth Conference Room on Tuesday, January 25, 2005. This talk introduced me, and the Continuous Assessment office (CAO). It discussed three types of educational needs and three corresponding types of educational software. Each helps track and assess students. Each allows manipulating and create reports from classroom data unobtrusively.

(*)WebWorks: Everpresent assessments that help students monitor their own mastery learning
(*) I(nteractive)Q(uestionnaire)s: Engagement in step-thinking, serious reading and writing and simplified grading of it all
(*) Interactive Portfolio Management: Organizational tools for involving students in projects or for having data about students ready and in report form All university people know that Learning and finding ways to use what we learn is tough.

WWs and IQs both illustrate many tools that Christy has brought up in our conversation:  step-thinking, scaffolding, guided practice, ZPD (zone of proximal development), and master learning.

IV. STUDENT PROFILE:

We also mentioned "knowing our audience". In other words, having an understanding of the 'student profiles' will be beneficial and provide a "value added" component to our outcomes.  At MSU-B the problem may be that we have to gather appropriate data for putting together student profiles to augment what they have here. A Summer Bridge Program would be a good start to knowing our students as they entered MSU-B.

We need access to serious data about students. For example, at UCI, when I had two years of Sloan Foundation funding for developing IQs, the registrar's office gave me my own account on their server, and I could look up any students' data. That told me a lot about my own classes when I curiously looked to compare performance with SAT scores. In this case, however, you should know I was trusted by many at UCI, including the Dean's office (of Physical Sciences), whose very bright staff person often consulted with me on such data prior to my being given this access. It was she who pointed out to me the value of such data.

In my paper on IQs, I have a quote from one of the students who responded so successfully to the support structure given by IQs. What made his success so astounding is that his SAT scores were not in the top 1/3rd of the class, the area at which I managed to get the full set of vector calculus fundamentals to work. Recall (?) the quote given by e-mail next quarter: "Dr. Fried, you wouldn't believe it. I showed the instructor how to do it [the problem]: You have to parametrize the set."

In the paper I couldn't say what were his SAT scores, but I knew them. That is why his case stood out so much for me, since success on this topic was achieved with 1/3rd of the class. That was only by using IQs. Traditional vector calculus classes, as reported by a committee at Berkeley, just gave up on it. 

V. MORE ON THE POTENTIAL FUNDING SOURCE:

Lastly, if we decide to pursue a NSF grant, the due date of Aug 2008 has been mentioned. I guess we need to determine our funding source....

Another program came across my desk from NSF: Interdisciplinary Training for Undergraduates in Biological and Mathematical Sciences (UBM) Program Solicitation NSF 08-510. It has a submission date in February, so I doubt we could possibly think of making that deadline. Still, a year from now, it is not out of the question.

Again, we are looking to prepare to go (by phone) to a granting officer and assure that we have a chance.