Finished another piece

This post is only to gloat over completing a sub-milestone on my PhD work, the second of the three analysis phases.




As the picture shows, I first did a set of six different kinds of analysis on each of the eight sessions I'm looking at for the thesis. I finished that about a month ago (after more than two years of work). The piece I just finished was comparing the questionnaires that I gave each practitioner to the individual session analyses.

I had asked each person questions about their software and facilitation skill levels and experience, and their opinions about the sessions they had conducted. There was nothing breathtaking that emerged from that analysis, but it filled in another piece of the puzzle by being able to compare what had happened in a session to the level of practitioner experience and skill.

For the last analysis sub-phase, "Cross-Episode Analysis", I will (at last!) look across all eight sessions and compare them on a number of criteria. I've been looking forward to doing this for years now, but had to finish all the individual analyses first.

Hoping to do this part in the next two months or less (I'll probably have to take some vacation time to do so). Then the actual writing of the thesis will start. It's been a long time coming (since Oct 2003).

The ethics of shaping

As the light at the end of the PhD tunnel starts to turn from a pinprick to a dime-shaped glow, several people that have recently listened to me talk about my research have mentioned that they see similarities in the work I do at my day job.

I work in software usability and user interface design for systems used by call center reps inside a large company. Like the practitioners I've been looking at for my research, making decisions about the UI in enterprise software design has the same degree of connections between choices about the form to give a screen and the way this will affect the interests of the people who come into contact with it. And, as with participatory representations, it's not just a unitary set of considerations. There are multiple kinds of people involved -- clients, end users, other IT teams, auditors -- each with diverse imperatives that drive them. Even "users" are not a monolithic block with one set of interests. What works best for an experienced, expert user is not the same as what would be best for a new user encountering a task or screen for the first time (among many other sorts of differences between users).

We are constantly balancing considerations like speed of development, ease of maintenance, testability, business rules, time constraints, future changes and plans, and usability. Each of these dimensions have ethical implications and trade-offs. As a usability person, it would of course be easiest to give ease of use the principal value (and of course, it's our job to do so) and give it the highest ethical importance. But if we know that the best UI design will have costs and cause problems for others with equally legitimate interests, we have to weigh those factors against others in our choice-making.

For example, take a simple change to an existing dialog box in an ordering system. In our work we get requests for these all the time, perhaps half a dozen a day (along with much larger projects). From a usability point of view, we always want dialog boxes to have clear window titles that give an overview of the situation; concise and straightforward text in the box that lets the user know why the box appeared and what they can or should do; and clearly labeled buttons that spell out the choices they can make (e.g. "Proceed with Order" and "Return to Address Entry" instead of "OK" and "Cancel" or "Yes" and "No", so often misused or unclear). N0-brainer, right?

In the abstract, yes. But what if the issue comes up (as they so often do) two days before code freeze for a complex release in which this change is just one of hundreds affecting many interdependent systems? And (as is often the case) what if the dialog box is in an older part of a system where all such boxes are coded as simple alerts using constants in a programming language which don't allow for descriptive button labels or window titles?

Insisting on "proper" usability design in this case would require custom programming, which takes time (that the development team doesn't have) and additional testing, which adds costs that comes out of someone's budget (which then is less available for other things). It may mean a change to documented requirements, which then requires formal review by people who already have not enough time for all they have to do, and creates risk to delivery which may jeopardize delivery and testing.

A simple answer in this case might be, "just make simple text changes using the existing code for this release, and do a proper job for the next release when there's more time". And often we do take that approach -- when we can. Sometimes there won't be another opportunity because the dialog box is in a part of the system that won't now get touched for a long time, and it would be too expensive to open the code just to make that change (since that, again, would require development time, testing, requirements documentation, etc.). So in many cases the changes are "now or never".

These kinds of dilemmas are very familiar to anyone working in software development, and I'm not saying anything new about them. However, from a research point of view I am particularly interested in highlighting how the choices that have to be made about something's visual form relate to the ethical dimensions of such choices -- the conflicting imperatives that are all valid and which all reflect legitimate interests.

My focus is on the ethical choices involved in making decisions about form, when in the context of shaping interaction and experience for others with mediating tools and representations. In my research I've been applying it to the special case of people playing a group facilitative role with hypermedia representations, but really it's more broadly applicable. What I feel emerging is a set of ways to enable other people to think and talk about these choices as they relate to their own work. I'm not so much interested in being the expert assessor myself, though I have had to do dozen of assessments of practice in the course of the research (I just finished the last of the 47 individual analyses yesterday!) and I do similar kinds of assessments every day on the job.

Rather, I want to enable and enhance people's ability to think about these issues for themselves, and give them some tools to do so. I want to make the question of "how does the way I shape the things I make affect the people who interact with them?" something that's accessible for people to talk about and get insight on.

Notes on Tim Small (part 3)

This is part 3, last of a series. Back to part 2

In this post I look at the ways Small applies his work to self- and peer- learner assessments, and how that might be useful for my research.

The practical application of the theoretical framework and analytical methods described in the article is in techniques for assessing learners that respect the continuum from aesthetic and experiential to efferent, without necessarily privileging one end of the spectrum. Small states that his approach "enables the products of enquiry-based learning and creative activity to be assessed objectively, but by criteria sympathetic to the experience they represent, rather than simply by arbitrary, external standards of accuracy, correctness or comparison." (p. 269)

This feels very much in accordance with where I am coming out with my research: a methodology for self- and peer- assessment of participatory representational practice, based on a set of values that foreground aesthetic experience and engagement (especially since they are usually passed over or marginalized), but don't leave out the "practical", "shared" and "constructive" aspects of working with groups in some sort of applied setting.

The main drive in Small's paper is developing criteria for learning assessment. Essentially, he proposes 'criteria for the criteria', providing a basis for the set of criteria he proposes. He looks for assessment criteria that, first, recognize and uphold the personal drive and responsibility of the learner (p. 261). For "learner" you can certainly substitute "practitioner." Second, the criteria should reflect the dynamic nature of the "journey" a learner experiences, between "self and text", "inner and outer", etc. For a practitioner, each session is in effect such a "journey", moving between (in varying ways) the aesthetic and efferent poles.

Third, he wants to give "safe passage" to the personal knowledge being created. In the analyses I've done to date, this part is perhaps less applicable, since practitioners in sessions are not there to develop their own personal knowledge, though they do draw on and perhaps build up such knowledge. However, this could be very applicable to the kind of self/peer assessment exercise that I took a first step toward at my IFVP session.

Finally, the criteria should respect the (quoting Fish 1980) "authority of the interpretive community". That part also is probably not as applicable to my research since the kind of practice I'm looking at isn't embedded in one particular institution or context. That is, there is not yet an existing "interpretive community", though I sure wish there was one. Helping to create such a community might be a possible future outcome.

In my research, I've come up with dimensions and criteria and observed them in practice, saying how they are exemplified in practitioner moves and choices. Is the next step to have these be self- and peer-assessed? As Small argues, engagement and commitment can only really be "accessed" through "subjective" self-evaluation (p. 268). Does this also mean that, for example, CEU can't really be assessed (different from "observed") except by either participants or practitioners themselves?

This conundrum does seem to keep recurring in some discussions of what I've done to date. What good does it do for an external observer (as I've been doing) to make assessments? Doesn't the whole point then hinge on how good the external observer is at making such assessments, rather than the value of the assessments, dimensions, and criteria themselves?

I don't want that to be the whole point. Does this argue that I should stop doing my own analyses and just move right to having people do them for themselves? Does it kind of negate some or much of the value of doing (and finishing!) the video analyses? Or can I legitimately say that I did the foundational work of discovering the dimensions and applying/iterating them, and the next step/future work is having people self-assess using them?

To move beyond this conundrum, I need to have practitioners apply my constructs to their own work for themselves, as the IFVP session was a first step towards. I may try to do at least one more such session before the final submission of my thesis. I do think that there is value for practitioners themselves to engage in such assessment exercises, if well facilitated. As Small puts it, "In involving learners systematically in reflection upon what they have achieved, we can ask them to question their own purpose, consider their stance, trace the movement of their selective attention, address their limitations and review their strategies." (p. 268) This kind of work will surely lead to improved practice, especially in the area of better understanding how one's actions as a practitioner affect the people one works with and for.

Last of a series. Tim Small's original article was published in the Curriculum Journal, Volume 20, Issue 3 September 2009 , pages 253 - 270.

Part 1
Part 2

Notes on Tim Small (part 2)

This is part 2 of a series. Back to part 1

In this post I focus on Tim Small's analytical methods and compare them to the approaches I've developed.

In his article, Small describes a method for analyzing students' responses to poetry that he developed for his doctoral thesis. He created a model for "analysis of written responses to poetry on the continuum between text and self" (p. 256), shown in Figure 1 below. It shows the dimensions of possible responses, ranging from the "efferent", outcome-based responses such as "spotting technical devices" or commenting on a poem's form, to aesthetic and experiential responses such as visual associations or even a student's making observations on the way they responded to the poem, such as "I was amazed how much my appreciation of the poem grew, the more I explored it." (p. 259).

Figure 1: Small's model for analyzing responses to poetry (Small p. 256)

He then applies this model to instances of actual responses to poetry by graphically mapping specific responses (taken from students' own writing about the poems) onto the model's dimensions, as shown in Figure 2:

Figure 2: An example of one of the analyses (Small p. 258)

This approach has a family resemblance to some of the analysis methods I've developed, particularly the "framing analysis" and the "CEU" tool. CEU diagrammatically represents the flow of practitioner/participant/representation coherence, engagement, and usefulness in a particpatory representation session, as seen in Figures 3 and 4:

Figure 3: Portion of a CEU analysis

Figure 4: Summary ("heat map") form of CEU analyses

The 3 dimensions of the CEU analysis are more limited than the 6 principal segments (with 24 sub-segments) of Small's model, which is more similar in form and intent to what I've called my "framing analysis" (Figure 5). I use this to characterize practitioner actions during a session in aesthetic, ethical, and experiential terms. It looks at how the practice and context interweave, and in what ways the aesthetic and ethical dimensions of the practice intertwine. I use this as the basis for a normative or ideal model to hold situations of practice up against. The model used in framing analysis provides a set of components, elements, and exploratory questions to help determine how a context of service, the unique set of people, goals, constraints, situation, and subject matter, can inform the "shaping" the practitioner performs on the representational object(s), and vice versa.

Figure 5: Components B and C of framing analysis model

I've applied the framing analysis in a textual rather than diagrammatic way (see an example in Figure 6), though it might be interesting to try that. I use it as a set of prompts or lenses on the way a practitioner acts in a session, answering the questions as a way to make sure I'm looking hard enough at what happened in a particular instance of practice.

Figure 6: Example of component A.2 from a framing analysis

Small justifies the analytical approach he took in his doctoral research by saying that the the findings themselves are tentative, but that it was concerned "as much with testing the model on a range of 'real' responses as with drawing firm conclusions from what it revealed." (p. 255) That resonates for me with how I am feeling about all the analysis of videos I've done. What is valuable about all that is not the hard and fast ideas of what constitutes 'good' practice or "lessons for practitioners", although there is certainly some of that there. Rather, doing all the cycles of analysis, reflection, and framework-building have been a testing and strengthening of the framework, finding in the data the constructs I've developed (or seeing ways to refine them).

What I've come up with is a way of looking at practice (as Small looked at poetry students) that highlight the aesthetic and ethical dimensions of the practitioner experience and a set of tools that provide representations of those dimensions, grounding them in repeated empirical analysis of actual practice. The question became "do I find what I think should be there" (and the answer I believe is "yes"). Now I'm beginning to see if practitioners themselves could do this kind of analysis of their own practice.

Some of these ideas make me think that I don't so much need to focus on "sensemaking moments" per se in my approach, but rather moments when the type and quality of attention and movement shift in some way from what came before (often, but not exclusively, in response to a sensemaking trigger).

Small provides a nice way to describe the potential value of analytical tools/visualizations of practice: "metaphorically creating a third dimension and offering a perspective from which to view the continuum" (p. 260), which offers "the means by with the reader gains access to a perspective from which to view the very response process he or she is engaged upon". This can result in "the mental space for viewing, understanding and communicating about the movement of his or her selective attention". Analytical artifacts, as "viewing instruments", create a "common vantage point ... for evidence to be included and shared in a peer or joint assessment dialogue." If and when I next try some of my tools out with practitioners evaluating their own practice (as the IFVP session was a first step towards), these considerations could provide a way to assess the success of the effort.

In the next and final part of this series, I'll look at the ways Small applies his work to self- and peer- learner assessments, and how that might be useful for my research.

Notes on Tim Small (part 1)

This is part 1 of a series.

I've recently spent some quality time with a paper Simon sent me: Tim Small's "Assessing enquiry-based learning: developing objective criteria from personal knowledge". Although Small writes primarily about student learners, there are many resonances with my research on participatory representation practitioners. In many places you could substitute "practitioner" for "learner" and the ideas work just as well.

Three dimensions of the paper are of most immediate value to me: Small's characterization of the experiential aspects of learning, particularly commitment and aesthetic engagement; his analytical methods; and the practical applications in the areas of self and peer assessment.

In this post I focus mostly on Small's ideas about engagement and commitment on the part of the learner, and the connections with practitioner experience as I have been looking at it. I'll look at the other dimensions in future posts.

Small provides helpful distinctions between the aesthetic ("lived-through experience" in the actual encounter) and the "efferent" (outcomes, "what you can carry away" p. 254) as two poles of a continuum. This idea of a continuum, rather than an either/or or absolute view, is just as applicable to participatory representational practice. The artifacts, methods, tools, and outcomes involved in such practices, important as they are, fall on the efferent end of the spectrum, but do not sum up all that is important about the practices; much of value lies at the other pole or in between.

Small quotes Rosenblatt: "Any literary transaction will fall somewhere in the continuum between the aesthetic and the efferent poles . . . there is a to and fro movement of attention between the words and the experienced, felt meaning being elicited, organised and reorganised." (p. 255) This concept of a transaction is similar to what I refer to as a practitioner move or choice. It maps quite well onto (for example) the coherence, engagement, and usefulness dimensions in my CEU tool, with "usefulness" at the efferent end of the scale, "engagement" at the aesthetic end, and "coherence" in between.

The way Small invokes Polanyi's idea of commitment in the context of developing criteria for learning assessment (p. 261) reminds me of Dewey's concept of "impulsion" in the artist's experience. "The task is to develop criteria that, first, recognise and uphold the drive and responsibility of the learner" (p. 261). Commitment involves a search for truth (discovery) and requires "strength of engagement" as well as "sensitivity" to self and text (read "representation"); a "readiness to interrogate", to "enter into a sort of dialogue with the text, object, or place." (pp 264-265). This is driven from within, finding expression in a kind of grappling with the stream of experience in order to make meaning: "A process of comprehending: a grasping of disjointed parts into a comprehensive whole." (p. 261, quoting Polanyi ("The Study of Man, 1959, p.28))

Real engagement of this sort, a willingness to be affected in some different or new way, "depends on the (at least partial) adoption of an aesthetic or personal stance... rather than simply the efferent stance of the analyst, who reads/learns according to a ''pre-ordained system of selection' (Rosenblatt 1978 p 89)". (p. 265)

The same kind of commitment, to make a session "work", to bring about positive outcomes, is in the practitioner and is something you can see or not see. To be a participatory representation practitioner in the kinds of settings I've looked at is to make a commitment, to live it for the duration of the role and the session. The overriding concern is how to make the session work. Not all the practitioners I've looked at were successful, but all did try to make generative and/or repairing moves when and how necessary. There could be some sort of measure or dimension of how far or how radical such moves had to be, just as another way to compare. The kinds of analysis I've been doing could be a measure of what practitioner commitment looked like and meant for that time (the session), what choices and trade-offs it occasioned, providing a context and a stable framework to describe them against.

The degree of engagement itself lies on a scale. While there is always a "transaction" (p. 255) between any aesthetic artifact and its audience, too often, even in participatory representation sessions, it's invisible, passive, or inconsequential. Like most movie watching, or walking by a painting hanging in a museum, most aesthetic encounters are superficial. Sometimes they aren't, and while that is sometimes a result of happenstance -- just being struck by something -- more often, it the encounter goes beyond the superficial it's because of some commitment on the part of the person -- some decision or desire (or classroom assignment) to make it mean more: "the commitment of the learner (to the truth he or she is seeking to discover)" (p. 263).

It reminds me of my college seminar on the films of Ingmar Bergman. I hadn't much liked the few Bergman movies I'd seen before the seminar, but I liked the professor and wanted to take the class. We had to see a film (this is in the stone age where the only way to see films was in a cinema, once, taking notes frantically in the dark, no pause or rewind) and write a 2-page paper due the next morning on some aspect. I wrote my first paper on "Wild Strawberries", basically scoffing at it. The professor said "no way" -- you can't take that superficial an approach -- you have to engage. That was sometimes painful (hard to *enjoy* a film like "Cries and Whispers") but it made me really pay attention and try to get at what was going on, and ended up transforming the way I looked at film and the whole question of the relationship of aesthetics and ethics.

In the case of participatory representational practice, for the practitioner (if not necessarily for the participants), that kind of commitment to engage has to be made; it's part and parcel, at least for the duration of the session. Part of that commitment is to get at least a partial engagement from the participants. What can practitioners do to foster, inculcate, and sustain this? To repair it in the face of disruptions? To improve the quality of participant engagement?

Small establishes the movement between inner/outer, subjective/objective, learner/teacher and talks about the "joint responsibility" (p. 268) they (should) have in assessment. That can be related to the putative "joint responsibility" of participants and practitioner in participatory representation construction. In the ideal mode, there is a joint responsibility and a movement between, a co-construction; it can even be said that it is the task of the practitioner to enable and foster that co-construction more than it is to create the artifact, or that the expertise of the practitioner should (only) lie in how good they are at creating the artifacts. Instead, the focus should be on creating the climate in which co-construction occurs, distinct from the 'same old thing' of normal meeting talk on the part of participants (however productive it might be; if the representation itself doesn't take on that which it needs to take on, it will not realize/contribute its full value), as well as all the work done by the practitioner him or herself on the representation (however well realized it might be).

Reframing practitioner and participant engagement in this way, in the context of participatory representation construction, leads to questions that can better characterize the specific role a representation can play in a session, and the relationship of practitioners and participants to it than taking a purely efferent approach. These include: in what ways does the artifact/representation speak for itself? What role does the artifact actually play, both in the here and now of the session, and later? What is the value of the artifact, proportional to coherence, engagement, usefulness etc.? (artifacts that aren't engaged in are akin to the well-written report that gets put on the shelf, or the graphic recording mural that no one looks at afterwards). How does one create situationally appropriate interventions, when the representation itself matters? What things do people do to make the session work? Why did they do them? What impact/effect did they have?

In the paper, Small makes a connection between aesthetics and ethics, at least where they meet in communication, and what I would see as the responsibility of practitioners to examine their relationships to representations and participants: "We might accept that a clearer understanding of the creative process would help us to get better at it, particularly if we were involved in communicating about our art, in teaching or being taught, for example. We might agree that it is the responsibility of all art with a communicative purpose to be self-referencing to a degree – even if only in stating its relation to its frame or context – and that that is what makes a work an objective entity, gives it its ‘comprehensive unity’." (p. 262)

Practitioner commitment requires awareness of, and sensitivity to aesthetic engagement, which (again resonating with Dewey) stresses being open to the uniqueness of each encounter (what Dewey calls "an experience"). Small writes: "Aesthetic engagement is denoted by a sensitivity both to text/object and self, combining two interactive functions: receptivity to the pattern of verbal and material symbols observable in and around the object/place, and subjectivity, the capacity for ‘infusing intellectual and emotional meanings’ into those symbols, (Rosenblatt 1968, 25) according to the ‘never-to-be-duplicated combination of attributes and circumstances’ brought by the reader/learner to each learning experience." (p. 265)

Nicely complementing the discussion of engagement (for my CEU purposes) is Small's discussion of "coherence and clarity": "the appropriateness of language, form, and structure to their purpose" as well as "completeness" and "persuasiveness", (p. 266)

More on the other dimensions of the article in these posts:
Part 2
Part 3

Reflections on IFVP session (pt 1)

This was meant to be the first in a series reflecting on a presentation I gave at the IFVP 2009 conference in Montréal a few months back. Though it's actually the second since I put up this post last week when I got the participant evaluations back.

The session, held on August 6, was the first chance to put the concepts and tools I've developed in my research in front of an audience of practitioners and see how well they landed.

Most of the preparation was done as I whiled away an evening in the dead end of La Guardia terminal B, concourse A, where all the Air Canada gates are. It's possibly the most run-down and crowded area of that airport, with the fewest amenities; not even a functioning water fountain. It was hot and stuffy and crowded, due to a number of weather delays that night. Huge floor-mounted portable air conditioners did little to cool the throngs, and the 2-foot-wide temporary ducts draped from the ceilings subtracted even further from the amount of free cubic space per person.

The pressure of work and family obligations in the weeks leading up to the session had not given me many opportunities to prepare. I'd thought that I would present the same material I'd given to several research audiences this year. But the flight delays gave me several precious hours of downtime to think.

Balancing my notebook on the hard bench arms, I began to realize that IFVP was a very different opportunity. With research audiences, you must establish bona fides on several levels and spend the time ensuring you are painting a picture that even researchers from different disciplines will accept, in terms of grounding in the literature, rigor of approach, etc.

But the IFVP audience (while it might very well include people with research backgrounds) did not come to Montréal to evaluate my research. Rather, they were there to take the next step in the development of their own practice and to strengthen their community. So this was both a challenge to me to lift my thinking away from research-presentational mode, but also to think about what I could say and do that would be of direct relevance and benefit to the IFVP practitioners.

As I mentioned in the previous post, I can't say I was completely successful in what I ended up doing. I ran short of time, both for the preparation and during the session itself. However, I did get closer to something that felt like a start in how to talk about and engage practitioners in this research in such a way as to benefit them as practitioners. It was a first step toward the next step, after I grind through the rest of my analysis, finish writing up, and defend the thesis. It is finding ways to provide value to just such communities of practice as IFVP that seems like it would validate all the time and effort I've put into this, that it was worth taking all this time to drive through the ambiguities and lassitude, the constant devil-on-the-shoulder of "why bother." At the end, there might be something of value to pass on to practitioners.

So, sitting in the Air Canada concourse, instead of just tweaking my usual set of slides, I wrote down what felt like the main things of importance I could say to this group. I wanted to talk about what is common to their practice and the one I've mostly studied, though even recently when I've lifted my head away from Compendium sessions I've started to see that there are all sorts of commonalities with any sort of participatory representational practice (such as the very different Visual Explorer method that I discussed here).

The first was that the choices we make as practitioners matter. We make many choices in the heat of the moment about how, when, and why to shape the representations we make, and our participants' interaction with the representations. In the course of sessions, we think -- very rapidly -- about ways to incorporate input from participants and respond to the constraints and events in the situation. Some of us combine verbal and physical kinds of facilitative interventions with the visual ones; some don't. There are other kinds of facilitative practice that emphasize spoken words, music, or other sorts of mediated discourse.

Making the choices involved in performing these sorts of practices visible has been the focus of my research. The choices and their consequences are not givens. They are situationally specific, so they have to be understood in context. Even with purely verbal kinds of facilitation or mediation, when it is done with people and involves the shaping of discourse, physical actions, as well as visual media, there are always both aesthetic and ethical dimensions in the choices we make, even if we're not consciously aware of them as we make them. The choices are there, and the aesthetic and ethical dimensions are there.

What I've done is develop some tools for how to look at and think about this kind of live practice. I've placed aesthetic and ethical choices at the center of my analysis, to "slow down the looking" so I can discern, first, what types of moves and choices are there. Second, what do the moves mean in the context they were made in. Even if the moves are canonical to some documented process, they are still choices being made by the practitioner (i.e., just because they're prescribed doesn't mean they're not active choices at the moment of performance). The moves shape the representation, and the interaction with the representation, toward some set of ends. What I've been pursuing is how to understand, describe, and characterize these in their context.

For IFVP, I was less interested in showing what I've seen when I've looked at Compendium practice (though possibly that might have been better, I avoided most of that because I didn't want people to get bogged down in trying to make sense of unfamiliar situations and tools, as has happened at times before) and more in seeing whether the analytical tools and ideas were useful for graphic facilitators to help think and talk about their own practice. That is really what I've been after all along -- not the classic kind of predictive hypotheses about what will and won't work better in the abstract (though people always seem to want this!), but rather to enable and expand reflective practice. I want to know, does this approach help you to reflect on your own practice? Specifically, do the ideas about aesthetics and shaping, ethics and consequences, give you helpful ways to talk to each other about what you do?

I gave a presentation talking about the above, and then with the (rapidly diminishing) time left, I set up a live instance of practice. The idea was to run the live session then have the participants use some of the tools and ideas I'd been talking about to reflect on what they'd just done. More on that in a future post.

Evaluations from the IFVP '09 session

Yesterday I got an email with scans of the evaluation forms that 22 participants at my IFVP session in Montreal in August filled out. They were mostly positive and/or constructive. A few were negative. These were mostly of the "this is too academic" variety, which is understandable.

I certainly could have done better with constructing the session itself, since I was still figuring out, up to the moment it started, how to best get across the material (and spirit) to an audience of practitioners rather than researchers. I spent more time than I realized (or wanted) on the presentation part before moving to the exercise part. This wasn't missed by the participants, and for any who may read this, my apologies. I learned quite a bit about how to do it better the next time I get such an opportunity (and I'm looking forward to doing this a lot more once I get the PhD done!).

Still, according to the comments, most felt there was some value in the session. I did some quick scoring. Of the 22 participants that completed an evaulation (there were about 40 in the room for the session), 15 (68.2%) were overall positive, 4 (18.2%) were at least partially positive, and 3 (13.6%) were clearly negative. Of the 80 total comments supplied to the 6 questions, 59 (73.8%) were positive, 6 (7.5%) were negative, and 15 (18.8%) were mixed or ambiguous. Excluding the 15 mixed/ambiguous, 90.8% were positive and 9.2% were negative.

More significantly for the research itself, though, it is pretty clear from the comments that the concepts resonated with the audience, which is encouraging.

Some examples:

• "He really has thought about the heart of our profession"
• "A new way to think about group work"
• "In every way -- This opens whole new worlds for me -- "
• "I liked the clarity of the heat maps"
• "Participatory practice could be a whole other domain to explore. This framework could be really interesting in educational settings!"
• "METAPHORS that emerged as the session/ideas unfolded: - the 'dance' between practitioner/participants; - the unfolding of a real time event as wild kayak paddle down a turbulent river"
• "Constantly linking aesthetics & ethics"
• "coherence, engagement + usefulness resonate as criteria that could be applied to many processes, i.e. teaching"

I have extensive notes from the exercise itself that I still need to fully analyze. More in a future post.

Thanks again to Jan Spaulding for putting together the evaluations.

Not really a dry spell

I haven't posted here in quite a while, but it's not because of lack of thought or activity. In fact the last few months have been very active, working on my analyses (almost done!) and quite a bit of synthesizing work associated with a) presenting at KMi in June, b) giving a talk and 'demo' of the research at the IFVP '09 conference in Montreal (more about that in another post), and c) responding to reviewer comments for the journal article we submitted for a special issue on creativity and design rationale. The latter (c) was the first time that I've gotten truly helpful feedback from a review process. All three reviews were positive (the most positive I've gotten, which is gratifying and makes me feel that I am actually making progress), but more to the point asked very thoughtful questions that are causing me to think carefully and, hopefully, clearly about a range of issues.

Sometimes I feel that all this should've happened say 20 years ago, not at my current more advanced age, but such is life. Better late than never.

One idea that occurred to me while working my way through the reviewer comments this morning (one of many such sessions in the last few weeks, deadline is end of Sept) is that many of the dimensions I'm putting at the center of my analysis -- narrative, sensemaking, creativity -- are essentially "recursive". Meaning, all of these can be looked at as both encompassing phenomena that we live "inside" of, as well as having to do with intentional acts and artifacts that we do or make. Some of the concepts that I need to clarify in the paper have to do with this -- making it clear to the reader which level of "recursion" I'm referring to. I drafted a table that shows all three concepts mapped onto both levels -- I think that will be helpful. It was helpful to me to draw it up, in any case. I'll put a version here later on.

So much of this work has to do with making distinctions that make it clear what you are and are not really talking about, it seems. I had put quite a bit of that in the first submitted version of the journal article, largely in response to past reviews that seemed to miss many of the points I was trying to make. I realized that this is the writer's, not the reader's, fault; I need to make the distinctions obvious and direct. Sometimes I wish it weren't so, that everyone would just know what I mean and respond to the ideas as I intend them, but also, such is life. Slowly I'm learning.

Compendium's software design, and argument mapping

This connects both to a recent post here, and to an email conversation between some of Compendium's inner core, where we've been debating some feature/design decisions. One hot topic has been the idea to retire the idea of IBIS / argument node types (Question, Idea, Pro, Con, Argument) in the default set. The debate reaches back to some of the early decisions and motivations when we created Compendium as a new software tool, which certainly took much of its basis from the desire to extend and customize Corporate Memory System's QuestMap, which was built around the IBIS rhetorical method. I thought I'd put some of my contentions here.

==

Compendium (as distinct from QuestMap) was designed from the start to allow different kinds of discourse to be interwoven with each other. Certainly affording, but not limited to -- or necessarily including -- argument mapping and IBIS.

What we have seen is that many of Compendium's users don't see a lot of the benefits and functionality of the software, in part because there are still remnants of the user interface that foreground IBIS, even though the rest of it has moved beyond that. People get hung up on the "argumentation" aspects, and if they are not interested in argumentation per se, they lose interest in Compendium.

On a software level, it also makes us preserve a lot of structure that really doesn't fit into the tool as designed/intended.

We want people to use Compendium for representing multiple ways of looking at issues, ideas, etc. If IBIS or other argumentation methods are part of their methodology (or if they discover IBIS on their own after adopting Compendium), the software should support it. But IBIS -- or any other single method or representation scheme, such as mindmapping (by far the majority of our downloaders that disclose their reason for interest in Compendium) -- should not be seen as the front door that potential adopters have to go through.
Users will come to IBIS if it makes sense to them (Jeff Conklin's work goes farther than anyone else's in that regard). But they won't use it just because the software shows them some node types that they may or may not use.

Part of what we've built into Compendium now is the ability to create branded versions with custom startup maps and other materials, so making an "IBIS Compendium" that looks like it is (only) an IBIS tool is now completely available. But many people want to use it for widely varying reasons, and we want to support that.

To me, the real brilliance of QuestMap as software was the ease and rapidity with which people could create and manipulate a hypermedia information space. What Compendium was about, from a software development point of view, was taking those hypermedia aspects and extending them in many directions.

From that point of view, IBIS, great as it is, is not a central aspect. It's transclusions (which we're going to rename to "embed" to catch up with the Web), templates, and tags, coupled with extensibility and customization, along with a host of new capabilities and concepts that have come from many different sources, which are central to Compendium as software.

Compendium is more than argumentation, or mind-mapping, or ...

Based on the discussions on the compendiuminstitute yahoogroup, and even more on the "nature of interest in Compendium" entries in our download log, there's been a surge of interest in Compendium in recent months. Much of this falls into two categories: people who are interested in Compendium's argument-mapping capabilities, or -- the larger category -- people who are investigating Compendium as an alternative to mind-mapping tools.

These are both great uses of Compendium, but there is a lot more to the software.

By intention, design, and functionality, Compendium provides many ways to link things together. You can start with making a single map that follows a particular scheme, such as IBIS or mind-mapping. As you build up more maps, you can use techniques like tags, transclusions, and templates to add many levels of connections. Compendium supports the creation of very large information spaces that can hold things like sets of pictures and photos, links to files, free-form text and writing, sound and movies, and tie them together in both formal (argument maps, models, structured views) and informal (let your imagination be the guide!) ways.

At its heart Compendium is a way to connect all sorts of views, ideas, images, and other resources together, in all sorts of ways. I encourage people to explore all the things they can do with it.