Post by Pavan Kadandale on May 22, 2015 15:17:36 GMT -8
Perhaps one of the larger impacts the LPSOE series has had on undergraduate education is the students' outlook and expectations for their time in college. We could design some survey questions about student outlook and expectations and administer them at different times/campuses and see whether having LPSOEs changes attitudes. Perhaps the same thing with faculty surveys to see whether faculty change the way they think about the classroom/education?
Another conjecture is that the existence of L(P)SOE faculty generally increases the amount of attention other faculty pay to teaching matters. This might be measurable by things such as self-reported number of conversations about teaching practices, number of teaching related workshops attended, number of curriculum updates considered by departments, number of faculty mentoring undergraduate research students, etc. The toughest part with these will be establishing baseline and getting critical mass of measurements. It also may be challenging to directly attribute observed changes to our existence, as opposed to general trends in education attention based on technology.
Post by CarrieMenke on May 24, 2015 19:29:47 GMT -8
Would students' outlook be largely influenced by demographics? If so, wouldn't that make it a bit difficult to establish an L(P)SOE-specific impact?
As for Pavan's suggestions, we could separate our impact compared to general trends by comparing L(P)SOE vs. non-L(P)SOE departments on the same campus as well as across campuses. (I don't have clear records of how many conversations I've had with my colleagues about teaching practices.)
Post by Pavan Kadandale on May 26, 2015 7:51:26 GMT -8
Sarah and Carrie, you both make excellent points.
1. I agree that student expectations, etc. might be determined by demographic factors - however, what we would be measuring is a change in these numbers, with at least some of that change attributable to the LPSOE series.
2. Self-reported faculty measures of the number of teaching-related conversations might be hard to interpret; but again, I think it's a great idea to incorporate as part of the entire package. To my mind, no one measure of "success" will be independently sufficient. I think it will be the impact we can show in a large number of related ways that will be most convincing. So, this would be one additional data point. We could also add some aspect of "learning about teaching methods" to this self reported measure, which might help us tease apart general conversations about education from what LPSOEs bring to the table.
I agree, the baseline will be the biggest issue. One thing we could do is use existing surveys, like this one that all graduating seniors take at UCI (http://www.oir.uci.edu/files/surveys/Senior-Surveys-Results.pdf).
There are general questions like:
Has your overall undergraduate experience at UCI made it more or less likely that you would attend graduate school?
Do you agree or disagree with this statement? "Looking back, I think the benefits I received by attending UCI are worth the financial cost to me and/or my family."
How well do you think your undergraduate experience at UCI has prepared you to acquire new skills and knowledge on your own?
I guess the question then is how do we convince people that positive changes are due to LPSOEs and not one of the other million things that change on campus from year to year. One thing is that we could get our campus to filter these by major I believe, so we could see is there a big chance in Bio Sci over the past 5 years (since our large cohort entered) versus somewhere like Physics which just got their first LPSOE.
While I agree that we should ultimately look at student outcomes, it may be difficult to collect this information without doing something like the Northwestern study, which compared different groups of faculty.
Instead, as a first pass, I think that we can look at departmental teaching "climate" (loosely defined). The results would likely be in the range of better to no change. There are some departmental "climate" surveys that are being developed, and I can try and look them up.
Another way to look at this is using Pavan's point #2 and develop a network analysis for who talks to whom about teaching. If we have departmental buy-in, this could be an annual survey in which everyone is asked to check off from a list of names all people with whom they had teaching conversations with in the past year. The survey would take fewer than 5 minutes, but the data could be phenomenal using network analysis.
Here are some hypotheses. The first one deals with general impact, and the last two deal with the potential contribution of teaching faculty. 1. Does network density (number of connections) increase over time after hiring teaching faculty and/or in departments with more teaching faculty? 2. Are teaching faculty more likely to be hubs or central actors in the networks? 3. If there are sub-networks within departments, are teaching faculty more likely to be liaisons who bridge multiple sub-networks ?
We can also look at more nuanced analyses such as people who teach the same course(s), how often do people talk about teaching, or the purposes of these conversations. Also, depending on the number of departments and their associated the variations, there may be interesting research questions that we can develop from comparing different types of networks.
Post by Pavan Kadandale on Jun 1, 2015 17:26:58 GMT -8
I think the network analysis idea is fantastic! We should definitely work on that; and that wouldn't require a pre- post-kind of study. We could just show where the L(P)SOEs slot into the teaching-communications network. That's awesome!
Post by Marina Crowder on Jun 8, 2015 8:24:36 GMT -8
All great ideas. As far as establishing a baseline for the impact of LPSOE's, specifically in BIOSCI, UCD may be a good place to look to. The first 'cohort' of LPSOE's have just been hired so there is an opportunity to survey both students and faculty prior to having LPSOE faculty in the department and then track specific impacts over time, such as student experience, faculty attention to teaching, changes in teaching methods used within the classroom, faculty participation in pedagogical development, etc.
I agree, using Davis as the "control" is ideal. I talked to Chris Pagliarulo (one of the directors of their I am STEM group) who will hopefully be at this fall's meeting about it. This means that we'll need to get some of our data collection instruments ready in the next couple months.
I also looked into the network analysis a little more. There's a CBE review here (http://www.lifescied.org/content/13/2/167.full.pdf+html) about it. This grad student actually presented a poster at SABER about a grad student network analysis he did. I didn't look at it that much, but I was thinking about contacting him to get some ideas for our faculty analysis. I think this would still be done in a pre-post way as well to see how the network changes when LPSOEs are added, although I agree the current picture will also be informative.
Ideally, I think that having longitudinal data (e.g. beginning of fall quarter every year) would allow us to examine changes over time in different departments and institutions with varying numbers or intensities of L(P)SOEs. Having a "pre" condition before the arrival of any L(P)SOEs is interesting, but we may not see meaningful changes until after a few years. We will probably have to collect data for a few years and then examine in a naturalistic way (kind of like natural history) to see how patterns and variations emerge. I think that multiple "post" points (e.g. annually) would be better than one "post" point.