Showing posts with label analysis. Show all posts
Showing posts with label analysis. Show all posts

Tuesday, 26 September 2023

Noteworthy disparities with four CAQDAS tools: explorations in organising live Twitter (now known as X) data

Written for researchers interested in extracting live X (formerly Twitter) data via Qualitative Data Analysis Software tools

Social Science Computer Review (SSRC) has just published a paper by yours truly, Dr Pat Harpur and Dr Corrie Uys to https://doi.org/10.1177/08944393231204163. As the article's title suggests, we focus on the contrasting the Qualitative Data Analysis Software (QDAS) packages that currently support live Twitter data imports. 

QDAS tools that support live data extraction are a relatively recent innovation. At the time of our fieldwork, four prominent QDAS provided this: only ATLAS.ti™, NVivo™, MAXQDA™ and QDA Miner™ had Twitter data import functionalities. Little has been written concerning the research implications of differences between their functionalities, and how such disparities might contribute to contrasting analytical opportunities. Consequently, early-stage researchers may experience difficulties in choosing an apt QDAS to extract live data for Twitter academic research.
In response to both methodological gaps, we spent almost a year working on a software comparison to address the research question (RQ) 'How do QDAS packages differ in what they offer for live Twitter data research during the organisational stage of qualitative analysis?'. Comparing their possible disparities seems worthwhile since what QDAS cannot, or poorly, support may strongly impact researchers’ microblogging data, its organisation, and scholars’ potential findings. In the preliminary phase of research, we developed a features checklist for each package, based on their online manuals, product descriptions and forum feedback related to live Twitter imports. This checklist confirmed wide-ranging disparities between QDAS, which were not unexpected since they are priced very differently- ranging from $600 for an ATLAS.ti subscription, to $3,650 for a QDAMiner (as part of the Provalis Research’s ProSuite package, which also includes WordStat 10 & Simstat).

To ensure that each week's Twitter data extractions could produce much data for potential evaluation, we focused on extracting and organising communiqués from the national electrical company, the Electricity Supply Commission (Eskom). ‘Load-shedding’ is the Pan South African Language Board’s word of the year for 2022 (PanSALB, 2022), due to it most frequent use in credible print, broadcast and online media. Invented as a euphemism by Eskom’s public-relations team, load-shedding describes electricity blackouts. Since 2007, planned rolling blackouts have been used in a rotating schedule for periods ‘where short supply threatens the integrity of the grid’ (McGregor & Nuttall, 2013). In the weeks up to, and during, the researchers’ fieldwork, Eskom, and the different stages of loadshedding strongly trended on Twitter. These tweets reflected the depth of public disapproval, discontent, anger, frustration, and general concern.

QDAS packages commonly serve as tools that researchers can use for four broad activities in the qualitative analysis process (Gilbert, Jackson, & di Gregorio, 2014). These are (a) organising- coding sets, families and hyperlinking; (b) exploring - models, maps, networks, coding and text searches; (c) reflecting - through memoing, annotating and mapping; and (d) integrating qualitative data through memoing with hyperlinks and merging projects (Davidson & di Gregorio, 2011; Di Gregorio, 2010; Lewins & Silver, 2014).
Notwithstanding the contrasts in the costs for different QDAS packages, it was still surprising how much the QDAS tools varied for the first activity, (a) ‘organising data’ in our qualitative research project: Notably, the quantum of data extracted for the same query differed, largely due to contrasts in the types and amount of data that the four QDAS could extract. Variations in how each supported visual organisation and thematic analysis also shaped researchers’ opportunities for becoming familiar with Twitter users and their tweet content. 
Such disparities suggest that choosing a suitable QDAS for organising live Twitter data must dovetail with a researcher’s focus: ATLAS.ti accommodates scholars focused on wrangling unstructured data for personal meaning-making, while MAXQDA suits the mixed-methods researcher. QDA Miner’s easy-to-learn user interface suits a highly efficient implementation of methods, whilst NVivo supports relatively rapid analysis of tweet content.
We hope that these findings might help guide Twitter social science researchers and others in QDAS tool selection. Our research has suggested recommendations for these tools developers to follow for potentially improving the user experience for Twitter researchers. Future research might explore disparities in other qualitative research phases, or contrast data extraction routes for a variety of microblogging services.  More broadly,  an opportunity for a methodological contribution exists regarding research that can define a strong rationale for the software comparison method.
The authors greatly appreciate the SSRC's editor, Professor Stephen Lyon, advice on improving our final manuscript. We also thank The Noakes Foundation for its grant AFSDV02- our interdisciplinary software comparison would not have been possible without funding to cover subscriptions to the most extensive versions of MAXQDA Analytics Pro and QDA Miner. All authors are affiliated with the Cape Peninsula University of Technology (CPUT) and appreciate CPUT's provision of licensed versions of ATLAS.ti.

Please comment below if you have any questions or comments regarding our paper?

Friday, 8 July 2016

Unexpected ethical challenges in using screen grabs of youths' #participatoryculture productions #visualresearch

Written primarily for researchers interested in the ethics of sharing young people's visual culture productions :


Advances in online image and text search may pose unexpected ethical challenges to researchers in protecting the privacy of their participants while sharing visual productions. I mistakenly assumed that depersonalising screen grab imagery would be sufficient to conceal teenagers' identities. However, in testing "depersonalised" screenshots of my participants' online portfolio screen grabs, I learnt that the ever-growing accuracy of text-and/or-image searches (i.e. via Google Image, TinEyeBing, Pinterest et al.) requires additional steps for dis-identification. Without these, sharing webpage screen grabs can potentially be used by undesirable audiences to locate young people's websites and contact details. Screen grabs may also pose reputation risks in potentially being shared long after participants might want them to be. Both types of risks will be weighed up against the benefits of sharing select students' e-portfolio productions in my thesis. These include visual representations making it easier for readers to become familiar with the online portfolio genre. Screen shots also provide visual support for research themes emerging from young people's choices.

Background to my visual research ethics challenge.

I had developed an original method for multimodal content analysis that used screen grabs to reverse-engineer the choices that 29 visual arts students made in using Carbonmade. To keep the rich nature of my visual data, I analysed these privately using NVivo. I then sought to de-identify select web page screenshots for sharing in conference presentations. I followed a process for visual anonymization, which was not extensive as I wanted to preserve most of the screen grab for accuracy. The anonymizing process involved Adobe Photoshop's blur function being used on several fields of every webpage. This ranged from the web address and portfolio name on every page to all mentions of their name on their profile pages and their contact details. It also involved checking that the e-portfolio's creator was not identifiable from their portrait picture and that no images disclosed their school's identity (i.e. school poster designs or uniforms). To further protect anonymity, image files were titled using pseudonyms.


"George", 'depersonalised' About page, 2012. A participant who gave permission for portfolio screen grabs to be shared. 
I also added select screen grab, two per A4 page, into my draft thesis's case study chapters. During their review, Associate Professor Marion Walton advised me to remove screen grabs that might expose its creators to ridicule and also to check the reverse search-ability of all images. She was concerned that these might not be truly anonymised. In checking, I learnt that the depersonalisation measures I took were insufficient. A 'visual specific dilemma' existed whereby my participants could still be traced through the following types of internet searches:

  1. An internet text search using text used in students’ self-descriptions under their About Us page; 
  2. An internet text search using the folder titles shown by the screenshots (i.e., in Google, using <e-portfolio software name> + <folder title>)
  3. An internet search using the image titles shown in the screenshots (i.e. in Google, using <e-portfolio software name> + <image title>)
  4. An internet image search using the screen grabs (for example in 'Google Images');
  5. An internet image search of the images inside the screenshots; 
  6. In addition, location information and other information in the case studies and school backgrounds could be used in narrowing image and text searches.

Testing the first four types revealed I had not successfully de-identified several screen grabs.

Ethical concerns and considerations.

This was concerning as it held ramifications for my future and past publications. It also had consequences for the ongoing e-portfolio pedagogy at the independent school research site:

I warned the e-portfolio educator, "Mr Proudfoot", that he should take additional steps to better protect student privacy via revising their e-portfolio pedagogy: my action research project found that teaching students to hide their contact details did provide a false sense of security, since teachers mistakenly believed that this made their students difficult to contact. Simply using students’ real names in online searches quickly served up their social network profiles. Some of these were public by default. Teachers must better support students with resources and examples of effective privacy protection that can at least minimise the dangers of ill-considered self-disclosure. This could include case studies of bad examples and in-depth advice on constructing pseudonymous personas. Schools should also provide support, such as policies and staff that young people can readily refer to in case of unsolicited online contact.

When my fieldwork began in 2010, I did not ask for student permission to use screen shots of their work. This was simply not a focus at the time. However, during my fieldwork I pioneered a screen grab analysis method that became heavily used in my 'Evidence' chapter. I also thought that screen grabs would prove helpful in adding a rich visual context to readers of my content analysis and eleven students' case studies.

I recently asked an ethics expert about protecting students' privacy and his advice contrasted to the cautious visual research feedback that I expected. He advised that since the screen shots are of web pages they are in the public domain already, I actually do not need these students' permission. Despite it not being a legal or institutional requirement, I remain mindful of the assurances that I gave to schools and students on protecting the research participants' privacy. Such assurances helped me overcome one challenge in securing ethics approval from the Western Cape Education Department/Department of Education and my two research sites. I am also aware that only a few of my case study subjects responded to Facebook or emailed requests for retro-active permission to publish anonymised screenshots in my thesis. 

My concerns around potential disclosure and lacking participants' explicit consent resonates with Prosser, Clark and Wiles' (2008) contention that concrete contextual issues and a researcher's individual moral framework must be added to legal and institutional requirements in making ethical visual research decisions. The risks to participants associated with disclosure may be small, but it does not sit well with my moral compass that the screen grabs in my thesis might provide visual evidence for subverting past assurances. Particularly now that the thesis itself is easy to source and search. In the past, the provision of UCT thesis hardcopies were mostly limited to its library. However, these are now automatically digitised for sharing post-graduation online via the library's website (and possibly Open UCT). Further, since I have already shared many screen grabs online in conference presentations, I must also explore reciprocal measures to protect my participants' privacy. For example, by replacing the screen grabs I shared in old presentations with properly anonymised ones.

To find out how other researchers have tackled the problem of depersonalising screen grabs, I did Google Scholar searches for guidance on anonymising 'screen grabs', 'screenshots' or 'screen captures'. I could not find relevant content, which seems to mirror the reality of screen capture techniques being mostly used for exemplars rather than in the research process itself.  Lacking a matching example to follow in visual culture research, I found Dr Kirsty Young's discussion of her research experiences with young people's online spaces (2013) particularly informative. It highlights several ethical dilemmas posed by new forms of research enabled by the internet.

My research project is unusual in being human subject research focused on public texts. It is the former as I have been involved in developing a new syllabus and doing face-to-face research with youths throughout e-portfolio lessons. However, I am also researching public texts since all my participants Carbonmade portfolios have no privacy restrictions. Given its unusual position in straddling both methods, I cannot expect unanimous agreement in the academic community regarding how the ethical principles of consent and anonymity pertain to my study. The public text argument versus one for the more onerous rules governing human subject research could easily be argued in both cases. This may pose unexpected problems for the publication of my visual research data. If research data cannot be shared it becomes redundant, which itself is unethical in wasting participants' time (Young, 2013).

In response, I must be cautious and take steps to ensure that my project's ethics in sharing screengrabs cannot be faulted from a human subject research perspective. While all participants and their parents/guardians consented to my research, some were only asked after my fieldwork concluded for permission to re-publish their work. I had not considered the future need to use young people's webpages publicly in academic publications. Given that the webpages are the intellectual property of their authors and that their content would be displayed more widely than the youth possibly intended, I intend to secure written consent for their academic use. This consent will address the timespan that informed consent is given for and afford options for the level of anonymity required. I will show my case study subjects examples of their dis-identified webpages to assist their decision-making.

Additional steps for depersonalising or anonymising screen grab images


Given the ready availability of image search sites and image reverse search applications, it is important for researchers to take steps to fully depersonalise images for participants' anonymity. As web page design is multimodal, it is also important that researchers filter both images and text. For example in my research into students' e-portfolios, I had to avoid mentioning folder titles verbatim in my thesis. I also must try to avoid quoting students’ profile descriptions verbatim for longer than three words.

The two alternate options (A - B) I tested for depersonalising screen capture images were:

A. Black out all text and replace profile image with silhouette outline

Option A. "George" depersonalised About page with with all text blacked out and profile image in silhouette outline, 2012


All text is blacked out, making it impossible for viewers to copy text strings in their searches. The blurred outline image is replaced with an outline drawing to add some visual information. 

B. Only add depersonalised screen grabs at small thumbnail sizes, organised inside tables

Option B. "George" de-personalised e-portfolio pages from 2012 reduced into thumbnail images in a table 


Here the size of each image is reduced to a thumbnail size for making their recognition via reverse image search more difficult. I tested each option in reverse image search engines and neither options A nor B produced results linked to its creator, let alone Carbonmade.

Google image search result for option A's image, 2016

Google image search result for option B's image, 2016

Both options enabled sufficient levels of anonymity in their results being linked to generic software entries. I then tested what would happen if a thumbnail image of student's work was selected from the table of thumbnail images. At such a small size, the highly-pixelated image results did not link back to their creator or Carbonmade during a reverse image search.

Google Images result for "George" depersonalised thumbnail painting crop 2016
While the process of dis-identifying over 80 images will be lengthy, I am pleased that I can use heavily anonymised imagery, rather than none. In addition to changing these thesis' images, I must also reciprocally update them in old presentations, which need then to be reloaded to Slideshare.

Request for comments... or turning this post into an academic paper.

This post was written to stimulate discussions on ethical issues related to the use of screen grabs.
It heeds the call to engage the general internet publishing publishing population in debates about the use of content for research purposes as this can ensure the ethical use of online content, (Young, 2013). Kindly add your thoughts by commenting below.

There is also a gap in the literature concerning ethical issues related to sharing screen grabs of young people's participatory culture as research evidence. If you would like this post to be upgraded into an article for helping close the gap, please get in touch. For updates on my research, follow this site or @travisnoakes.

Total pageviews since 2008's launch =

+ TRANSLATE

> Translate posts into your preferred language

+ SEARCH

> Search travisnoakes.co.za

+ or search by labels (keywords)

research (56) education (43) design (22) nvivo (16) multimodal (9) visual culture (4)

+ or search blogposts by date

+ FOLLOW
+ RELATED ONLINE PRESENCES

> Tweets

> Kudos

> ResearchGate profile
Articles + chapters

> Web of Science


> Social bookmarks + Edublogs listing
diigo education pioneer Find this blog in the education blogs directory

> Pinterest
> Create With Pinterest