An honest look at participatory design and user involvement

A refreshingly honest article about the realities of participatory design and involving users in systems design projects. It accords with my experiences as an IT/usability/PD practitioner for the last two decades.

"It is time to speak honestly about the gap between our intentions to build working systems and our ability to do so in practice. This gap is typically not caused by a lack of effort on behalf of developers or users, but rather is the result of misdirected efforts. The systems development and implementation process will continue to be overly challenging if we work against the tide by trying to make users fit our theories of how and when they should participate in development initiatives. Instead, we suggest catching waves with users at opportune moments, working to hear what they are saying, and then adjusting our, and their, expectations about when a system is completed." (p. 55)

As the authors say, it's not a matter of not including and involving users -- that has to be done. My own design practices revolve around it. What it's saying is that doing so is not a panacea and does not in and of itself guarantee you any special insight or improved chance at success. You still have to interpret, be flexible, do different things at different times, improvise and make sense of what you are hearing, how you are listening, what the users themselves can and do pay attention to, etc. Doing so is also an ethical stance for design practitioners -- no matter how good, they aren't going to know and perfectly understand in advance what user input means and what has to be done.

The article is Wagner, E. L. & Piccoli, G. (2007), 'Moving beyond user participation to achieve successful IS design', Communications of the ACM 50(12); pp.51-55.

Canny and uncanny people

Something quite compelling in this statement, when reflecting on the ups and downs of pursuing a large, lonely research project:

"Canny people tend to succeed in their own lifetimes; uncanny people tend to be recognized and appreciated only centuries later, because during their time their actions appeared to be either insane or pointless."

From "The new dynamics of strategy: Sense-making in a complex and complicated world" by C. F. Kurtz and D. J. Snowden, IBM Systems Journal, Vol 42, No 3, 2003

One can reassure oneself, during the downs -- the times when one indeed feels that one's efforts are both insane and pointless -- with the idea that one is, in fact, uncanny. Yeah, that's the ticket.

Human Technology journal article now live

In the Special Issue on Creativity and Rationale in Software Design
John M. Carroll, Guest Editor


Our article is: The Practice Level in Participatory Design Rationale: Studying Practitioner Moves and Choices

Albert M. Selvin, Simon J. Buckingham Shum and Mark Aakhus

Unity of purpose, communication, and representation

Further on the idea of integral representations and the role of practitioners in making representations matter...


As I was working through writing up the "how good/successful was the session" question, it struck me that one way to characterize this was by considering how closely three dimensions were unified in each session -- the purpose, or intended (as well as emergent) reasons or goals for each session; the communication, both verbal and non-verbal, the way the participants (as well as practitioners) interacted, and the representation, the visual artifact (Compendium maps in the cases I'm studying, but could be any kind of visual or written representation, or even just a verbal representation if there is some sense of a central focus for the session):



The best sessions see a fusion of the three dimensions. Purpose, communication, and representation become indistinguishable in practice; why we’re doing what we’re doing, how we interact with each other, and the visual artifact appear to be unified:



By contrast, in less optimal sessions, the dimensions are disconnected from each other. For example, people could be talking about something that is more or less removed from the ostensible purpose of the session, and the representation itself is ignored or irrelevant:

Integral representations

As I've been working through writing up the comparative analyses, one thing that's struck me is that a key differentiator of practice styles and expertise is how much (and in what ways) practitioners make the representations themselves matter to the participants and to the proceedings, as opposed to being (in varying ways) a sideshow, background, or decoration.

For example, one of the Shaping aspects in Category B (practitioner interaction with participants) is the "degree of intervention to get participants to look at the representation". It occurred to me that, while all of the studied practitioners used various physical and verbal means to do this, the style, purpose, and strength with which they did this varied greatly. Both of the 'expert' sessions (Hab Crew and Remote Science Team) did this frequently and in depth throughout the session, but also did it with a degree of naturalness. They did not have to use much special force or emphasis because, due in large part to their expertise, the representation was integral to the proceesings, embedded in how and why the group was working.

This isn't something that comes for free and takes a lot of factors to achieve, but the phrase integral representations (seemingly widely used in mathematics) seems to me to sum up a lot of what this research is aiming toward: how to make such representations integral to their participants and audience, how to make them matter, and in what ways.

And the final 'raw' cross-session analysis artifacts

Cross-session Shaping comparisons (table version) (Compendium version)

Cross-session Framing comparisons (table version) (Compendium version)

CEU comparisons

Some 'raw' comparisons of the "CEU" (Coherence, Engagement, and Usefulness) analyses across sessions are here.

Next and final step for the comparative analysis -- create a summary of the cross-session Framing analysis.

Comparative framing analysis

Compendium version here.

One more piece of cross-session analysis to go -- CEU comparisons. I hope to finish those tomorrow, then it's on to start writing up the first draft of the analysis chapter.

Comparative analysis of sensemaking moments

A fuller writeup later, but in case you would like to look:

In Compendium form

In summary tables

Preliminary results from cross-session 'shaping' analysis

I've finished the first round of comparative analysis (earlier steps described here). This morning I translated the mapping of each session along the 29 shaping dimensions into Excel. It's not a perfect or exhaustive study (that would take another set of years) but did yield some interesting comparisons.

In the mapping linked above, I'd ranked or grouped each of the 8 sessions for each of the dimensions, adding rationale for each ranking and sometimes example artifacts, such as screenshots, photos, or transcript quotes.

The Excel work was originally to see if any patterns emerged by comparing across the dimensions. I think some are. At least it gives some compelling visuals.

In the spreadsheet, for each dimension, I arrayed the sessions left to right, assigning a unique color to each (blue for Ames Group 1 (AG1), orange for the Hab Crew session (Hab), etc.). Where the dimensions were more simple groupings (such as dimension A1, which just shows the practitioner choice of method), I indicate that rather than a high-to-low array. There is a lot more info and description of each dimension and ranking method in the maps.

Group A are 5 dimensions concerning aspects of a session's pre-existing plan and other pre-session factors such as choice of method and approach. The results look like this:

The 9 Group B dimensions cover aspects of practitioner interaction with participants:

Group C comprises 5 dimensions showing various aspects of the sessions as meetings and discussions:



And Group D is 8 dimensions that focus more specifically on the shaping of the representation:
What was most interesting about moving the analysis into Excel was better being able to compare the dimensions themselves and think about what they might be showing as a whole. It's still early going but it does seem like there are some interesting patterns from a participatory shaping point of view.

For example, it occurred to me that some of the dimensions that came out of the data group into categories that could be used as a kind of index of 'goodness' at least from a shaping point of view. I looked over all the dimensions and identified these 13 that could contribute to such an index:

• A5: Degree of practitioner adherence to the intended method during the session
• A6: Participant adherence/faithfulness to the intended plan
• B8: Practitioner willingness to intervene – frequency and depth of intervention
• B10: Degree of practitioner-asked clarifying questions to participant input
• B11: Degree which practitioners requested validation of changes to representation
• B13: Degree of intervention to get participants to look at the representation
• B14: Degree of collaboration between multiple practitioners (if applicable)
• B15: Degree of collaboration/co-construction between practitioners and participants
• C17: How “good”/successful was the session?
• D22: How much attention to textual refinement of shaping
• D23: How much attention to visual/spatial refinement of shaping
• D24: How much attention to hypertextual refinement of shaping
• D25: Degree of ‘finishedness’ of the artifacts
  

I then derived "scores" for each session along these dimensions by assigning 8 points for the highest-ranked session in a dimension, 7 points for 2nd place, etc. Of course these ratings are largely subjective, but in aggregate and for comparison reasons I think they have some validity (and at least, I've captured the rationale for each rating I gave, so someone else could look at the data and understand (if not agree with) why I ranked them this way).

By doing this, an overall ranking of sessions along this "shaping index" emerges. As you can see in this table, not surprisingly the Hab Crew and Remote Science team sessions are at the top (since they had the most experienced practitioners and were in situ sessions drawn from actual projects). But the Rutgers and Ames sessions also array along this meta-dimension. I then derived a "software proficiency rank" and a "facilitation proficiency rank" from the questionnaire data to compare to the shaping index, and (even to my statistically untutored eye) there appear to be some good correlations, especially with facilitation proficiency.


Here are graphical views of the (self-reported) facilitation proficiency and software proficiency comparison scores:


This made me think there are more interesting comparisons to be derived from the other composite comparisons from the questionnaire data. I could probably spend months going through all the possibilities:


However, I don't want to drown in all this data or let it take me too far off course of really getting at the aesthetics, ethics, and experience aspects of these sessions. Next I will be looking at the sensemaking moment analyses from a comparative perspective, and see what that yields.