Formative interventions and design research
Discussions of method don’t often sound all that sexy, but I love them anyway. My first two sessions of the ICLS 2008 conference have been about method (maybe process is a better term). First, Yrjo Engestrom talked about formative interventions, an activity theory-style approach to research, and then Ilya Zitter described her process for using Educational Design Research in her doctoral work.
Not surprisingly, Engestrom railed against the “gold standard” of randomized controlled trials as the best and only way to properly conduct research. He mixed in a couple jabs at the U.S. – one for emphasizing such studies and one for making unpopular interventions. I’m with him on both. Randomized controlled trials (RCT) shouldn’t be the gold standard for all kinds of research, and the U.S. shouldn’t have intervened in Iraq. At least not the way we did. But, I digress. I was talking about method.
In contrast to the positivist RCT program, Engestrom recommends a different process entirely. His process, we’ll call it formative interventions (that was on his slides), engages the research site as a participant in the project rather than as a passive recipient of a designed intervention. It differs from ICT (and even from Design Research – an approach gaining popularity in education research) in three main ways:
- starting point,
- process, and
- outcome.
The starting point for formative interventions are poorly understood objects. RCT and design research start with some goal in mind. Having a goal presupposes that the goal is desirable. I dislike the arrogance behind starting a project from, “I know how it should be,” and so it’s no surprise that I like formative interventions’ starting point.
Engestrom calls formative interventions’ process “double stimulation.” That term doesn’t really work for me. I think what he means is that the research introduces and recognizes changes in the research environment over time. Whether those changes are planned by the researchers or not is not terribly important. The process of studying a changing phenomenon differs dramatically from the “execute, refine, repeat” approach RCT takes.
Lastly, the outcomes of the two methodological approaches differs. For Engestrom, the outcome should be “new activity concepts” and for RCT, it’s “solutions.” I’m often frustrated by “solution” terminology – because I’m uncomfortable labeling social phenomena as broken, because I’ve seen too many “solutions” that don’t have clear “problems”, because I just don’t see the world that black and white.
So now we have an outline of Engestrom’s preferred methodological approach. I like it. It’s engaged, rigorous, and embraces the ongoing and changing nature of social situations. Trouble is, it’s hard to sell, in the U.S. especially, and even harder to do.
Enter Ilya Zitter. Ilya is a PhD student at Utrecht University, and she uses a method she calls “Educational Design Research” in her doctoral work. Basically, she uses research, design, and practice approaches to study undergrads in a projects course. Hooray for higher education at ICLS! It’s almost as satisfying for me to engage as adults’ informal and workplace learning. Anyway, Ilya gave a short talk in a firehose session where she described how she conducted her research. This is exactly the kind of talk I like to attend at conferences. I can read papers, but papers about how the research was conducted are hard to come by. Sure, papers include methods sections, but those don’t often tell you the nitty gritty details. Ilya talked about her struggle to balance research, design, and practice in her work. This is a struggle I get to avoid in my dissertation but which is central to my life at Microsoft Research.
At MSR, we’re engaged in a formative intervention study of sorts. We’re working with HR and managers to adjust social and technological tools used in onboarding at Microsoft. I’m often uncomfortable in the “design” and “intervene” portions of such studies. I much prefer to be a fly on the wall. That’s not immediately useful (or publishable) though. I, like Ilya, am struggling to find balance and to negotiate relationships among researchers and practitioners all while gathering and analyzing data. It’s hard, but at least I’m not alone.
What is an actor?
Some colleagues and I recently submitted a paper to a conference, and last week I sent in our rebuttals to the reviewers' comments. Our paper introduces some terms from actor-network theory (ANT) to an audience that isn't terribly familiar with ANT. I like ANT as a method, not really a theory, for helping sort through really dense, unfamiliar data. For instance, you can use ANT to help you figure out where to focus. If you enter a scenario as an ignorant sponge (as many qualitative methods ask that you do), it can be difficult to figure out what's important. It's also impossible to pay attention to everything all the time. ANT can help you find some important actors on which to focus your attention. Actors seems like a familiar term - we know of many in Hollywood, we understand what it means to act even off screen. That's not what ANT means though. For ANT, actors are something like things that cause change, or things that other actors say are actors. You can probably see why some of our reviewers got a bit confused.
I use ANT in my dissertation to talk about what had to happen for a specific bridge to be built. I couch the study in terms of actors who did work to produce the bridge. I borrow actors from ANT in that I consider non-human actors (e.g. bendable concrete) symmetrical with human actors. Objects and ideas can do work even though they're not human. They're identified as important by other actors. For instance, using bendable concrete in the bridge deck required changes in how the sidewalk was connected, how the deck connected to the regular concrete deck on either side. The bendable concrete was acting in that it was creating change. Other actors, such as a construction consultant, identified it as an actor by saying things like, "If we use that bendable concrete, then we can't use rebar there. We'll have to use something else." In that excerpt, he identified bendable concrete as the thing that caused a change. Bendable concrete has some agency. Had we entered the construction project without knowing anything, we'd know from the way the consultant talks about the bendable concrete that it is something important, that determines what other actors may or may not do (e.g. use rebar).
I think our paper does a good job of describing how ANT can help identify the important things in a set of data. When I find out if it got accepted, I'll blog about the conference itself. I have pretty strong feelings about the conference to which we submitted, and they will either grow stronger or remain in check, depending on the outcome of our submission. Oh, that drama! The intrigue! Stay tuned.
Blogging on the bus
It occurs to me, while riding the Microsoft Connector bus (MSFT's private, wireless-enabled buses that shuttle us through a reverse commute between Seattle and Redmond) back to my sublet, that I have changed. When you read what I have to say, you may think that only what I think has changed and that I remain the same. I assure you that’s not the case, but that rather, I have changed, my very being is different now than it was. First, I’ll tell you how I know this is true. I prefer Ann Arbor to Seattle.
That’s right, I admit that I prefer Ann Arbor to a bustling city more than twice its size and complete with public transit, professional sports, excellent restaurants, and a slew of other things I’ve been missing for years. The trouble is, working a “real” job with real hours means I don’t have the time or energy to enjoy these Seattle offerings. At the end of a day like today, what I most want to do is to sit in my yard or one of my friends’ yards, drink a beer, grill some meat, and talk about nothing and everything. If I were really lucky, it would be a grilling night at Bill and Jolie’s or I'd be sitting outside at Zingerman’s.
Ann Arbor and Seattle are the sum of their parts, and right now, I miss Ann Arbor’s parts. I miss my roommates, my yard, my friends, my cats, my home office, my 10 minute commute, my kitchen, my chef’s knife, my fellow social scientists. Sure, Seattle has fresh food, hiking, Microsoft, old friends I haven’t seen in a year or longer, poker rooms, a new and interesting research project, and many other things to recommend it. The trouble is, I made a home in Ann Arbor. I thought I’d made some friends and rented a house, but I made a home. And I miss it.
A new vocabulary
I'm lucky to be the kind of researcher I am. I get to observe and interview people who do really cool work and to learn about what they do. A couple years ago I learned how vaccines for Black Plague get made. My dissertation lets me learn about how bridges (real ones, not just metaphorical ones) get built. Now, at Microsoft, I learn how software is built. Over the last couple of weeks I've been interviewing managers of developers and testers at Microsoft in an effort to recruit them for my study on remote onboarding and to learn about what they do.
Years ago, before I came back to grad school, I had the illustrious title "Developer" at a web start up in Chicago with about 100 employees. A "software team" there was a project manager, a developer, an architect, and a designer. We built websites. My job was to write ASP code that made the designers and project managers happy. Towards the end of my career, I wrote ASP.NET code. Somewhat more complicated, still produced a website. "Developer" at Microsoft means something a bit different. Developers at Microsoft build stuff that matters - Windows, for example. They do it in teams using tools such as Source Depot, Razzle, test harnesses, RSOPs, WTT, and TFS. They meet in scrums, war rooms, Live Meetings, Office Communicator, one-on-ones, and code reviews. Those 12 phrases and acronyms are new to me. Not one of them had I ever heard before. I now know what 5 of them mean. I'll leave you to guess which 5 I know.
Learning the vocabulary of my subjects is just one part of my research, but it's been a while since I had so much specialized vocabulary to learn. The phrases and acronyms the engineers I study use seem a bit more intuitive to me, things like "pancake test" and "aggregate" are nearly self-explanatory. Granted, "code review" means about what you think it does but "scrum"? No, developers are not playing rugby.
Being a new employee while studying new employees is so meta I can hardly handle it. Perhaps next week when I meet my first new employee subjects I'll start to feel like I have a better handle on the situation. For now, while I'm meeting with managers, I'll just keep typing as fast as I can and hope that I'll know when to ask for help.
Learning about visas
I'm just starting my third week as an intern at Microsoft Research, and I'm still figuring out how to blog this experience. I'm working in the Human Interactions in Programming group studying remote onboarding of new employees in the Microsoft Canada Development Centre (MCDC).
We originally defined remote onboarding as a process new hires who are geographically separated from their teams go through when joining a new company. After interviewing managers and HR professionals, it makes more sense to think of remote onboarding as a process the organization goes through to help new employees be productive. We use remote instead of distributed because only one person is physically separate from the rest (sometimes called a "one-off"). For me, distributed refers to a group whose members are in a number of different places, either together in groups or apart in one-offs or groups. Basically remote is a subcategory of distributed and a special enough category to get its own name. Microsoft uses a similar term, remote management, to refer to the kind of management leads and other managers must use to work with employees who are far from them, whether at MCDC or in India, Ireland, China, etc.
I'm conducting a comparative case study in order to understand how remote onboarding works and how various interventions impact onboarding experiences. Of course some part of my energy is directed at identifying areas for growth so that Microsoft can improve their onboarding, but it's too early in the study for me to talk about improvements. That said, I think I'm ready to say the U.S. has some serious room for improvement in its visa and immigration rules.
Many people are working from MCDC while they await an H1-B visa. Others are planning to stay in Canada for some time. A third group are waiting for L visas. As I understand it, both H1-B and L visas are work visas; people who hold them are able to work in the U.S. H1-B's are the visas awarded through lotteries that we hear and read about while Congress and the Presidential candidates debate immigration reform. Microsoft has been pretty open about its feelings about immigration laws. They want to hire more foreign workers because they are qualified, but the U.S. won't let them in the country. Enter MCDC. Canada apparently likes the idea of highly-skilled workers with good salaries living within its borders. L visas are internal transfer visas and are not part of a lottery system. Basically, if you work for Microsoft in another country for 365 days, you can then get an internal transfer and L visa to come live in the U.S. and work for Microsoft for 5 years. The H and L visas differ in their rules for getting them, the rights you and your family members have in the U.S., the length of stay, renewal, etc. I'll be learning all about visas in the next couple of months. I'm pretty sure I'll think we need some reform though. 65,000 H1-B's clearly aren't the right answer to the global competitiveness challenge.