Feedback from readers

Feedback from readers of 'How to read a paper'

I find it somewhat ironic that I have received more feedback on this non-technical paper than for all of my other research put together. Here is anonymized feedback from readers that may be useful to other readers.

---

I've been interested in the process of learning (in particular my own, which I feel can sometimes be dull, so I wanted to know what was it that makes learning exciting for me (do we have to put up with all the dullness? Is there necessary and unnecessary dullness, just like necessary and unnecessary complication?)) and came across your paper on "How to read a paper". It seems to make explicit some things I have started to do, like exploding references from multiple papers and collecting the best ones. But you also say to look at the conferences where the key researchers have published, which is something I haven't done yet. It's interesting how we (still have to) do this by hand and how Google and such tools are only part of the iterative 'search process' (whereas some people call it quits with a single search, especially in day to day life). I love the idea of making a (personal) virtual reconstruction and comparing it to the author's presentation. And the idea of a living document. Generally, I have a sense that learning can be so much more interactive, individual, local, as opposed to forcing oneself to be satisfied with a mass-produced generalised material. And yet we are told that everything is done for us, what with automation and whatnot. But it is so exciting to think that indeed, not everything is done for us, not everything can be found without some effort, search and re-search (again search, iterative search (is that not an pleonasm?))...

And that the reader reads in different modes e.g. at night when tired (but you don't mention in the morning when overly excited and apt to burn oneself out! or perhaps this doesn't happen). And the reader's iterative interaction and implicit understandings and contexts and purposes and needs. That reading and learning, even/especially about computers, is a human process. It's all so wonderful.

Anyways, now I have to explode "you", and all you've written about communication and humans and computers ... all that has ever been written ... what a daunting task. Can it ever be completed? Perhaps a local completion of sorts, appropriate to a time and place.

...

--

P.S. by explode I mean, generate, (as in generate and prune)...

P.P.S. the chinese have a very concise way of making 'references', in the specific situation of explaining which word they are using (since there are many homophones), especially when telling someone their name. For example, the surname might be "白" (white, pronounced "bai"), so a person would say "蒼白的白“ ("cangbai de bai"), meaning "the bai which is part of the two-character word cangbai"). I suppose this would be similar to when we say something like "Wadler's symmetric calculus". It's just interesting that this referencing has to go on automatically in an everyday situation in Chinese. I suppose it has to do with the ambiguity. Perhaps we can call Chinese an 'even more natural language than English', if by naturality we mean ambiguity.

---

"I am a PhD student in Economics and my biggest struggle is literature reviews. Since there is always a flood of work to review and it takes tremendous time and effort just to skim the papers I am usually lost and discouraged even in this seemingly easy stage of the research. I was telling myself I have to come up with a way to set a limit to the papers I need to read, but I was not able to stop myself from downloading papers as I encounter them. I still need to do a lot of practice with your suggested way of reviewing literature. But I already have a relief in my mind with the limit it sets to the papers to be downloaded. "

---

"...while I do a literature survey and I am looking for a basic principle it helps to read papers not just in my field of specialization but in other fields where the same principles are used. It brings a very different perspective and sometimes opens up new methods of thinking. "

---

"My comment is this; intuitively, I have been following something like the same technique for some time and have found it very useful. However, the one thing you didn't elaborate on is time-scale. I find the process works best when there is quite a large amount of time between each stage of review. For example, I find that I often collect large numbers of papers, read them (10 minute rapid first review) and then come back to them sometimes weeks later for a second pass. Finally, some further weeks or months later I return to them and find that I can extract the final useful insight that I had not done previously. (This also works really well for reading large PhD-thesis type documents).

Obviously, this might not be useful for reviewing papers under a tight deadline, but as a PhD student reading an extremely large number of papers from very diverse fields (e.g., psychology, neurology, psychoacoustics, etc), I find this very useful."

---

"You may also ask to print out the papers only after the first pass, if the reader select that paper for second pass. Because I have seen that many people just print out papers and through them away just after reading the abstract. I do not see any difficulty of going through first pass on the screen, without printing it out. It will save some more trees."

---

"In the 2.1 section "The first pass", the second "C"---Context, which is in the five "Cs", the paper says "Which other papers is it related to? Which theoretical bases were used to analyze the problem?" I am not sure if the word "which" could be used properly like this when "other papers" and "theoretical bases" are both plurals. I have tried to search the keywords of "which" combined with "is it related to" and found most of results are in the form of "Which A is it related..." where A is grammatically singular. But lots of results of "What Bs is it related to" also showed up, where Bs is plural. Would it be possibly that the word "What" instead of "Which" is the correction expression?"

---

"Loved it, though I have another document that I read in the past that helped me tremendously as well.

http://cseweb.ucsd.edu/~wgg/CSE210/howtoread.html "

---

"One suggestion: the version I have (from here http://www.cs.stonybrook.edu/~vyas/teaching/CSE_592/Fall12/papers/howtoread.pdf) has an unwieldy length of 2.25 pages. From my perspective, the paper would be more accessible, especially as a reference material, if it were only 2 sides (so I can print it as one page, double-sided). Also, there is something unsatisfactory about that extra quarter side."

---

"However, I have some problems in the third pass. You mentioned 'to attempt to virtually re-implement the paper'. I don't quite understand the concept 'virtually'. Could you explain a little more? Also, re-implementing the work is so tough that it seems impossible. For example, how to construct the author's environment to do experiments?"

---

"Happy to contact through mail. It is to appreciate you for giving such a paper. Am pursuing my Ph.D i was asking many people how to write a paper.... but their answers were not fulfilling me.

i was keep on searching the net for my question. suddenly in some website i have seen the link "how to read a paper" i clicked i git your paper.. before and all i just like that read papers without any formality...but now i came to know with clear format and clarity after reading yours.

you have specified three approaches nice sir. thank you somuch... Now-a-days my view on reading the papers has been totally changed. It is worthful..

---

Moreover, besides Google Scholar and CiteSeer, I recommend libra.msra.cn as another way to quickly find top conferences, top papers and top institutes in a specific area. I tried it in the field of Computer Graphics, and it works well. DBLP is also a good website for academic research. It stores indices of huge numbers of papers and updates quickly.

---

In fact, I worked on my own version of "How to Read a Paper" on my own blog ( http://systemsci.org/jinshanw/) right after I got a msg from my colleague on the existence of your article and before I read it. I then demonstrated "how to read a paper" using your article as an example applying my own steps about reading a paper, which is very similar with yours but with some subtle difference.

First, about the format of your article, I would like to suggest to put every important step in lists. Since you used a list to express steps in the first pass, also since I am a lazy man with more or less my own ideas on this already, while I was reading your article, I wished that in the second and the third pass you have also used lists. That way, I can almost avoid finding important inform from usual texts. I agree with you that formulea, figures, conclusion are more important than other buried lines. I suggest to add "lists" to this list of important formats. Also keeping format consistent (in texts on all of your three passes in the three sections) makes it easier to read.

Second, about the contents, I would like to suggest you to add my step 0. While supervising my own students, I notice basic of the field including terms, tastes, feelings and directions, are more important than others in determining whether or not a student can understand or appreciate a paper. What I refer to in step 0 is that some basic things beyond textbooks "but before" research papers. I guess before a student start to read a research paper, he/she is more or less familiar with the basic textbook level concepts and skills already. However, between textbooks and research books there is a gap, and quite often it is this gap making reading papers hard to some students. What I suggest is that every time this happens, then one can start from a review paper. A good review paper will provide a bridge for crossing this gap. Besides a review paper, maybe getting some feelings, directions, not only knowledge and skills, from their supervisors so that one can be motivated and can improve ones taste better, is also important.

On all other issues, I agree with you 100 percents, which makes my steps almost a plagiarized version of yours, ^_^. Some further details are written in Chinese and I guess they will stay in Chinese for a long time since I am such a lazy person. But, hey, laziness is one of the motivation of progress of science and technology, right?

At last, I would like to suggest to publishers (or the author himself/herself) that: it will be much better if when a paper is published, a table of contents and also a concept map explaining important concepts and major contributions of the work are published together with the paper. BTW, this is related to what I am working on currently: knowledge management, education, statistical physics and network science.

Anyway, good work and I am happy to see one more teacher who cares about students and tried to help them out, and who also share very similar thoughts on "how to read a paper". On a second thought, I should probability spend the time writing to you this email on translating the blog article in English, ^_^.

---

I suggest though that you give the reader that is new to proper reading an idea of the kind of things you write in the margins of a paper. Today while reading your paper I wrote in the margins for the first time in my reading life. I wrote the definitions of the words 'prototype' and 'implicit' which I had come across several times in my reading but not bothered to find out what they really meant. That is an example of what I wrote in the margins. May be if you gave the reader some of yours it would be very helpful in their future reading.

---

A few months ago, I published my first paper and wrote my PhD project to apply for a grant to undertake it. Luckily, my director helped me a lot but the process of writing was so hard (science speaks English...I'm from Argentina, so language is another challenge) that definitely I NEED to find a method.

I like the "Three-pass approach", I will apply it. Another difficult part for me is the next step: to compare and connect conclusions, facts and concepts from differents papers. When you deal with many papers, how do you organize all that information?. I'd like to know if you have a method to do it. It should be simple: to focus on the most important concepts and conclusions to understand all points of view about the topic.

Writing a short summary in my own words or keywords of the main concepts from each paper are both good ideas, but I´d like to know your opinion. Will you recommend bibliography for me to read?. I really want to master this skill.

---

I have few comments about this paper, of course, maybe they are not correct, just from myself prospective.

1) Could you add a table to summary the three-pass method？ like give a word to define each pass, what’s purpose for each pass? which sections do we need to read in each pass? take how long ? how to deal with reference in each pass? Contrast them will give more impression and understanding to readers 2) Could you add a pseudo-code for how to proceed this method? Because three-pass method has sort of logic relationship from start, like maybe continue or stop, like put it away for a while in order to cold treatment and then continue to read to refresh minds 3) Give a case study could be more prefect

---

The Open University would very much like permission to use your paper – HOW TO READ A PAPER in OpenLearn. OpenLearn is part of the OU’s widening participation objectives. This initiative places chunks of its courses online and open to all. The initiative recognises that the ability to learn and intelligence exists across all communities and OpenLearn provides the opportunity for people to learn and test their learning experience thro’ OpenLearn without pressure. It also supports professional development in this way. The learning units provide a complete teaching and learning experience.

To that end the OU (Dr Lucia Rapanotti – Snr Lecturer in our Dept of Computing and Communications) would like to keep intact to the unit taken from M813 (Computer Development) and your article.

This unit when published may also be of use to your own institution as it will be available to all.

---

Two months ago, I read your paper ‘how to read a paper’. I try to conduct it according to the content when I read the papers in my domain. It is very helpful, by doing pass 1, I could have an outline of one paper; pass2 makes me grasp the main content in one paper; and pass 3 tells me how to analysis a paper in depth.

However, I still encounter some problems in order to read paper more efficiently and smartly.

First, in pass 1, you suggest to find ‘assumptions’ and ‘theoretical bases’. I realize that assumption is very essential, because in pass 3, you suggest to ‘virtually re-implement’/’re-create ’ the work. But I am always confused about ‘assumptions’ even after finish pass 2. What are the assumptions in one paper, and how to grasp it exactly? I think that maybe I just ignore it. As my simple opinion, for example, there is new idea in one paper, is this idea a big assumption? If in depth, how to resolve it to more assumptions in details?

Pass 1 just has 5-10 minutes duration as you said, after this quick review, why we could decide that this paper is well written or not? What are the criterions? Is it the length, number of graphics, or mathematical functions?

---

The following reference URL seems to be broken: [3] G.M. Whitesides, “Whitesides’ Group: Writing a Paper,” http://www.che.iitm.ac.in/misc/dd/writepaper.pdf

This one works for me: http://onlinelibrary.wiley.com/doi/10.1002/adma.200400767/pdf

---

In your paper, you have given some figures for the time it must take to read each of the 3 passes, for beginners and pros. I think it is absolutely necessary that you have put these time figures. The reason it is necessary in my opinion is that it gives a specific and measurable goal, that one can roughly compare his progress against. True, it's not precise, but it is still something. It gives some information, rather than not giving any (As is unfortunately norm with almost all authors who --conservatively-- don't give any information in similar situations)

Question: 1. What is the time expectation for reading papers in my field (as described above)? 2. It's probably a delicate technique to realize if one has to retreat to reading background, before putting any much more effort into the paper. a. What are the signs that one must switch to reading background? b. How much background (and how deep) should one really read for the task of accomplishing the paper reading task, as all textbooks are 500+ pages and they take a life time to read fully.

---

During the first months in my PhD I felt lost in reading papers, and wasting time reading incorrect articles, your methodology is interesting, I was using only two steps but the 3 steps reading will give me more analyse in my work. Moreover I didn't know the times, by trial and error my beginning performance was 5 hours per paper, right now between 2-3 hours depending of the journal.

I would like to suggest the next points for future publications:

1) Nowadays, researchers are using Mendeley or Endnote to manage journal articles, when I read I use Mendeley to write annotations and after write my literature review.

2) Optimum time a researcher can read, pauses, when we have to start writing the ideas after reading one paper or maybe 5 papers in the same field, is there a methodology?.

I will do a research of the Related Work section in your paper, I'm interested to improve my efficiency, you can imagine the difficulty for an international student (my language is spanish) to read and write scientifically in English.

Kind regards from Townsville, Australia.

---

It's number 6 here: http://cstheory.stackexchange.com/questions/1168/what-papers-should-everyone-read

---

However, I must point out that you have apparently neglected to describe the technique which I personally have found most effective in my several decades of experience reading papers. To wit, printing the paper or papers out on physical paper, stapling their pages together, and carrying them with me to a trendy sidewalk cafe in an upscale part of town, and them reading them while ordering 500 ml servings of dark Belgian beer.

When accompanied by an ink pen and a yellow highlighter marker, I have found that this technique greatly increases my comprehension, satisfaction, and the chances that I can make meaningful edits to Wikipedia based on the papers without being reverted. Please share the benefits of your thoughts and ideas on these matters with me.

(My response: Caffeine >> Beer)

After extensive field trials, I have found that your proposed improvement has yielded mixed results.

In general, I find that my accuracy and initial throughput are increased with the application of caffeine, but long term throughput has suffered due to motivation issues which are not present when ethanol is applied.

In order to complete the 2 x 2 experimental matrix I shall henceforth utilize Irish coffee, and report back with the results after a sufficient number of trials.

---

I wish I had read this back in 2009 when I was formulating my PhD thesis topic! I wanted to tell you one particular mistake I did back then, which resulted in lot of wasted time. If you think this is useful, perhaps you can comment a little bit on this in your article to save some other poor souls in the same situation.

First off, I have to admit that I had trouble finding my research topic. So, I resorted to reading a lot of papers in the hope of finding some idea that will extend a previous idea or provide some kind of counter solution. I would easily take about half-a-day or more to completely read and understand a single research paper (I did not follow your three phase approach, but I would read the abstract to see if the paper I am about to read is a relevant paper). After reading the paper I would look at the Related Work section and start the process all over again. I did this for several months, But unfortunately for me, this excessive paper reading didn't help me find my research topic. When I finished reading one paper, I had collected about another dozen or so in my to-read paper stack. Eventually this process became too tiring and frustrating for me, and I gave up reading papers to find my research topic all together.

So, for someone who's hopeful about finding new research frontiers and advancing the state-of-the-art, would you recommend this approach? Would contacting the authors of the papers help? Or should you reconstruct their work and see how you can expand on it? I didn't do any of those, but I think I should have paused for a moment between each paper I thought was pretty good and initiate connections or experimented myself on the ideas presented in the paper.

---

the "how to read a paper" paper is quite helpful. it's a very disciplined way of getting the most out a given amount of time. i certainly follow some of the advice when looking over papers, for example the ones from a faculty candidate, but not in an organized way, so this will help me do a better job.

one thing you don't mention is whether you think it worthwhile to make some kind of summary, perhaps after the second pass. i find it helpful to write down a short paragraph that summarizes the paper; even if i subsequently lose the summary, the act of writing is a help for remembering things later on. i do this most often with student independent work and theses, but with referee reports as well. it's a supplement to your suggestion of making notes in the margin, which is also very useful.

---

I came across your paper on papers (!) andit reminded me of a couple of things in the first pass stage - moreover it put them in a logic that was similar to how I was already doing it so I've incorporated them into my matrix.

It's currently on the home page of my website: http://psychologyinc.blogspot.com with a reference to your paper.

The reason I send this, other than the obvious, is that you note in your paper you want it to be a living document. If there's anything that my matrix has that may be of use to your work feel free to take it and improve on it as you wish.

---

I have just read your manuscript on how to read a paper. You requested feedback. My only addition to your document would be the usefulness of citation managers when references begin to number ten or more. I prefer RefWorks and Zotero, but End Note is good as well. I enjoyed reading your work.

---

I had come across your article on "How to read a paper" which was written for the 37th Volume of ACM SIGCOMM Computer Communication Review during July 2007.

The Students Association of Physics Department publishes "Psi-Phy" (http://www.phy.iitb.ac.in/doku/doku.php/sapd/psi-phy) - an annual student's magazine. The magazine tries, among other things, to provide useful academic information to the students of Engineering Physics. Your article would be an excellent aid for students to their junior or senior thesis and the article would greatly benefit all students.

We would be very grateful if we could include your article in our magazine. We shall include the appropriate acknowledgements.

---

Thanks for the excellent info provided in How to Read a Paper. I linked the article on my new site Knowledgesummit.net. The article appears in the Other Intellecutal Reads section. If you have a chance check out the site www.knowledgesummit.net and any feedback would be appreciated. The site is in it's early stages will improve in the future. Below is a brief intro to the site. Please forward to anyone who may be interested.

Knowledge Summit What to Expect www.knowledgesummit.net

I am happy to say it is finally here. I have been playing with the idea of designing a site that addresses all things science, logic, philosophy, and some general interest topics. The site is finally ready. The site will be updated on a regular basis (every few days). I am confident the site will be a hit and further readers Knowledge on important subject matter and arm them with the knowledge to dismiss much of the misleading information we are subjected to on a daily basis

---

I use a very similar approach myself, but there were gray areas (especially when I wanted to explain it to my students). This is very concise and clear, and very useful. I also have a comment that you might find useful. One thing that I always struggled with was archiving the papers I have read, keeping track of notes, searching them, .... I have tried various tools for this, and different approaches, but have found a new tool that is absolutely a delight to work with. It's a free Firefox add-on called Zotero (zotero.org). It imports and exports many bibliography formats (including Bibtex), has the capability of storing the original paper along with the bib entry, add notes, tag, very nice search tools, ... and it knows ACM and IEEE portals (i.e. once you do a search in your Firefox browser, it can imports them to your collection). You can drag and drop your references (say to an e-mail, with the style that you choose, say IEEE). It also has a MS Word plug-in for those who prefer that. I don't know if you have tried it before or not, but if not, I would highly recommend it.

---

I've just finished reading your excellent paper on How to Read a Paper and would like to make a complaint. Why didn't you write this paper 3 years ago? It would have saved me a great deal of anguish whilst struggling to plough my way through stodgy texts that weren't really relevant to my interests!

-)

I've only happened across your article by chance - my course is in Information & Library Studies but I work in a computing library so came across your paper while indexing the ACM Computer Communication Review. Do you plan to publish this in any other journals? I'm sure that it would be useful to students in many different fields.

I liked your paper for a couple of reasons - firstly, it encourages critical reading and makes the point that if a paper is not relevant or if the research is poorly executed then don't waste time on it. I think it's hard for inexperienced researchers to have the confidence to do this so your advice is sound.

Secondly, it is good that you include time-scales in how long each pass should take. I remember feeling anxiety at how long it takes to read a paper properly and it is useful to know that even experienced researchers can take hours to read a paper.

Thanks for your article. I'll be recommending it to everyone I know!

---

Background: I've been in the process of writing a paper on evolutionary computing for a few months now, and due to being interrupted by other work, I realized that I forgot a lot about the papers I already read through once I return to paper-writing. I don't take printouts (save trees), so I decided to keep a spreadsheet file in Google Sheets where I type the name of the paper in one column and my notes/comments in the corresponding columns. This helps quickly refresh my memory when I return back to my research. Having it in Google Sheets also helps me review it on my mobile phone when I'm travelling.

Suggestion: In your paper also, you've mentioned writing notes during the "second pass", but the wording assumes that the person has taken a printout and makes notes at the edge of the page. If you feel my points (mentioned below) are significant enough to be mentioned in your paper, it would be an honor for me to have contributed.

My points:

1. The sentence "It helps to jot down the key points, or to make comments in the margins" could be altered to something like "Whether you review a single paper or multiple papers, it helps to maintain a document either online or offline, where you jot down the name of the paper and your comments about it. The comments could pertain to a summary of how the paper is relevant to your research, terms you didn't understand or questions you may want to ask the author".

2. It may help to have a bit more emphasis on the necessity of maintaining a document that contains the researcher's observations about multiple papers, especially when a person is in the process of writing a survey paper or when a person is multi-tasking, gets distracted frequently and needs to return to the work.

---