Let’s discuss!
Reflections on helping organise the 4th Asia Pacific Conference on Educational Integrity, Wollongong, Australia, 28–30 September 2009, with a focus on discussions and papers
Brian Martin
Arts Faculty
University of Wollongong
bmartin@uow.edu.au
29 March 2010
At most conferences, contributors present papers and audiences listen. Some of us on the conference organising committee wanted a different approach. We had been to plenty of conferences where we were bored to exhaustion listening to papers. Some speakers are good; others are tedious. The main point is that being a passive listener soon becomes tiring.
Discussion time is almost always the most stimulating part of a conference session. Presenters are supposed to keep to a time limit to allow time for questions, but many go over their limit so that there’s time for just a couple of questions, or maybe not any. Sometimes question time becomes an extension of the talk: someone asks a question and the speaker takes five minutes responding and introducing new points.
If discussion time is the most stimulating part of a session, why not make it the main part? Some of us had experienced different conference styles, so we thought we would try to turn the sessions into discussions. To facilitate this, papers needed to be written and distributed beforehand.
Ideally, every person present would have read the paper beforehand, but we knew this would never happen. We needed a way to allow all participants to have a basic grasp of the issues. We came up with the idea of a 1-pager: one page of information about the paper, including an abstract, a list of key ideas, and two discussion questions, all fitting on one page.
We asked authors to submit a 1-pager months before the conference. Writing the discussion questions would encourage them to think “it’s a discussion, not a presentation.” The full papers were due six weeks before the conference. We put all the papers — with the first page of each one being the 1-pager — on a website and gave all participants the site address two weeks before the conference.
At many conferences, organisers dread drop-outs: authors who cancel at the last minute, potentially throwing the programme into chaos. By basing the sessions on written papers, we avoided this problem entirely. Discussions could proceed even without the author present. As it turned out, authors of two papers for our conference were unable to attend, but it didn’t cause any problem for anyone else.
Discussing written papers also allowed an innovation: anonymous papers. A few authors didn’t want their identity revealed, because what they were saying was provocative. Only one or two members of our organising committee knew the identity of these authors, who may or may not have been in attendance at the conference itself.
We didn’t expect everyone at a session to have read the full paper — the 1-pager was the substitute — but we did hope that several participants would have read the full paper, enabling some depth in the discussion. The assist in this, we came up with the idea of a discussant: someone — not the author — who would lead off the discussion and add a discussion question. The discussant would be expected to read the paper, of course.
We wanted a genuine discussion involving as many participants who wanted to contribute. We anticipated the main obstacle would be authors who reverted to the usual mode of giving a talk and others who fostered this by asking questions of the author. So we added to the mix what we called a moderator, namely someone to facilitate the discussion and make sure it focussed on the discussion questions rather than on the author. Each session started out with comments from the discussant, followed by comments from the author (if present) and then general discussion.
To foster a smooth discussion process, we wrote notes for moderators, discussants, authors and other participants (see Appendix 1) and ran two workshops for potential moderators in which we discussed likely problems and ran some role-plays.
Overall, the re-orientation to discussions rather than presentations worked extremely well. Changing the terminology was a challenge. In our committee, we had to continually correct each other whenever someone referred to a presentation or a speaker, emphasising our terms discussion and author. At the conference itself, most authors adapted to the role of mostly being a listener rather than the centre of attention.
A week before the conference, we invited 37 individuals to be discussants — one for each paper — usually inviting another author who was likely to have some interest in the topic. We chose discussants who were at a different university than the author. Nearly everyone approached was willing and all discussants took their roles seriously. Our conclusion: having discussants is a good way to generate more depth in the discussion. In some cases, discussants and authors had in-depth discussions outside their sessions.
For making the sessions run well, the role of the moderator is crucial. The eight of us on the organising committee, plus one other local academic, were the moderators, and every one of us participated in the moderator training. We each moderated at least four sessions. It wasn’t always easy keeping the conversation on topic and enabling everyone to contribute.
Discussions usually worked best with 8 to 12 participants. Some groups were smaller than this; those discussions usually worked all right too. The biggest challenge was large groups, over 15 people. If everyone spoke to the maximum time of two minutes, the half-hour session would be over without any interaction. One option was to hear from the discussant and author and then break into two or three sub-groups, with one discussion question assigned to each sub-group. This wasn’t ideal — an author could be in only one sub-group and missed out on other groups — but enabled better interaction. We could have prepared for this by having more than one trained moderator in sessions likely to attract large numbers.
The papers
For the way we ran the conference, we knew written papers were essential. We wanted to have enough of them to have plenty of interesting discussions. There were several decisions to make.
When would the papers be due? We decided that full papers would be due six weeks before the conference, with the 1-pagers due a few months before this. Overall, this worked well. The 1-pagers were more of a commitment than a typical conference abstract. With only a couple of exceptions, everyone who produced a 1-pager produced a paper. We ended up with 37 papers in all.
We learned that most authors operate to deadlines. Only one paper arrived really early, along with the 1-pager; most of them arrived within a few days of the deadline. To encourage authors to complete their papers, we sent out reminders a month and a fortnight before the deadline.
Should the papers be refereed? Many academics — certainly in Australia — need their papers to be refereed to obtain formal credit through the publication certification system. So we decided that we should have an option for papers to be refereed. But would papers need to be refereed to be suitable vehicles for discussions? We decided that non-refereed papers would be fine. In fact, short and provocative papers, without references, can work quite well for fostering discussions though they wouldn’t pass muster with referees. We set a flexible paper length to cater for different styles: 1000 to 10,000 words, with any self-consistent referencing system allowed.
Trying to get papers refereed before the conference seemed an enormous task, so we decided on a flexible approach. Authors could ask for refereeing on submission. Another option, to encourage those who had written 1-pagers to write full papers, we said that they could write a short preliminary version of their papers for the conference and, if they so decided, prepare a fuller paper for refereeing afterwards. This, we think, encouraged some authors to write papers when otherwise they might have dropped out.
Refereeing is seen as a guarantee of quality, but we had something else to foster quality even in non-refereed papers: all of them would be posted on the web in the conference proceedings. For most authors, having one’s work openly accessible to peers is a strong incentive to make the paper coherent and sensible. With presentations, it’s possible to get by with minimal preparation and only a few in the audience will be aware of the shortcomings. Open publication offers a level of ongoing scrutiny that is more likely to discourage slapdash efforts — and so it turned out.
For papers whose authors requested refereeing, we decided to obtain reports from at least two referees and, in most cases, to avoid using referees from the University of Wollongong. Because no one on the organising committee was a referee for any pape, we could discuss potential referees more freely.
As the deadline for written papers approached, we realised a problem for our chosen system of double-blind refereeing. The papers needed to be on the web for conference participants to download and read — but surely this would reveal the authors’ identities to referees. We came up with a solution: we put the papers on a temporary website to which there were no links, so it would not be found by search engines. In particular, there was no link from the conference website to the papers. Only conference participants would receive the web address for the papers, so referees wouldn’t know about it. As an extra layer of protection, we omitted authors’ names from online versions of papers being refereed and listed only titles in the conference programme.
Of course, when participants arrived at the conference, they found out who was author of which paper. But this information was unlikely to leak out to referees. Our system did mean that after the conference began, we couldn’t use participants as referees.
To get away from the typical experience of overly critical referee reports, we used a supportive approach of asking for positive comments and ways to improve papers — see Appendix 2. Quite a few referees said they appreciated this approach; the result was more encouraging for authors, though some of the changes requested were quite extensive.
In summary, the most crucial decisions for our approach to the conference were:
• having discussions of written papers
• training moderators to facilitate discussions
• having instructions for all participants about how the discussions would operate
• having a range of options for written papers: long and short; formal and less formal; refereed and non-refereed; refereed before or after the conference.
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Appendix 1
Notes for discussants, authors, moderators and other participants 4th Asia Pacific Conference on Educational Integrity Wollongong, 28–30 September 2009.
Discussants
Read the paper beforehand!
The author posed two discussion questions. You get to present a third one. Write it on the board.
You get to open the session with a comment. Aim to speak 1 to 2 minutes and be graceful when you’re cut off at 3 minutes.
Read the notes for moderators and try to assist the moderator.
Authors
The discussant will open the session. Then you will have an opportunity to reply or comment. Aim to speak 1 to 2 minutes and be graceful when you’re cut off at 3 minutes.
You’re not giving a talk — you’re participating in a discussion. You’ve had your say in your paper. Try to spend most of the session listening.
You might find it useful to keep a record of key points raised during the session. You can take notes yourself or ask another participant to do this for you.
Read the notes for moderators and try to assist the moderator, most likely by being exceedingly brief and encouraging others to comment.
Moderators
General principles
Your goal as moderator is to foster a stimulating discussion.
You should aim to give everyone an opportunity to speak.
You should attempt to prevent anyone dominating or hogging the discussion.
You should encourage expression of diverse viewpoints. You should try to keep the discussion focused on the discussion questions.
Your role should primarily be moderating the discussion, not contributing to it. If you have a lot you want to say about the topic, get someone else to be the moderator.
Suggestions
As people arrive, make sure everyone has a 1-pager.
Begin by inviting each participant to introduce themselves — names and affiliations only. Introduce yourself first.
Have a timer with a buzzer. (You can be the timekeeper or have someone else do it.)
Introduce the discussant. Time: 3 minutes maximum. Be firm.
Author response: 3 minutes maximum. Be firm.
Subsequent contributions to discussion: 2 minutes maximum per person.
An optional procedure after the discussant and author’s opening comments: go around the group, inviting each person to comment for up to one minute.
When some participants begin to take a second turn, ask those who haven’t spoken yet if they want to say anything. Pay special attention to limiting contributions by the author. Try to prevent the session becoming question-and-answer.
Keep the focus on the discussion questions.
At the end of the session, thank the author, discussant and all participants.
Other participantsRead the 1-pager. If possible, read the paper. Be prepared to offer a comment, but don’t feel obliged. Read the notes for moderators and try to assist the moderator.
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Appendix 2
Typical letter to referees
Dear xxx,
I invite you to referee the attached paper, "Title-of-paper." I thought of you because one of your publications is cited in the paper [or some other reason].
The paper is for the 4th Asia Pacific Conference on Educational Integrity, to be held here in Wollongong on 28-30 September this year (see http://www.uow.edu.au/conferences/4APCEI_2009/home.html). I'm the editor of the conference proceedings.
We are running the conference somewhat unusually this year. Authors will not present papers. Instead, papers are put on the web beforehand and sessions will involve facilitated discussion on questions prepared by the author and a discussant.
For refereeing, I'm also adopting a somewhat unusual approach. What I'd like you to send me are just two things:
1. Good things about the paper
2. Specific ways the paper could be improved
That means no criticisms; these can be reformulated as ways to improve the paper.
You can make a recommendation about acceptance of the paper if you want, but it's not necessary. For my approach to writing a helpful referee's report, see http://www.bmartin.cc/pubs/08jspwhrr.html.
You can be anonymous or not, as you prefer.
If you're not able to write a report, just let me know. If you are able, as I hope, then I'd appreciate receiving it in 4 weeks - or even sooner!
Regards,
Brian
Brian Martin
Faculty of Arts
University of Wollongong, NSW 2522, Australia
phone: +61-2-4228 7860 (home), +61-2-4221 3763 (work)
fax: +61-2-4221 5341
email: bmartin@uow.edu.au
web: http://www.bmartin.cc/
Subscribe to RSS Feed (Opens in New Window)