A methodology for Artificial Intelligence Reviews (AIR)

Original contribution by Rugare Mugumbate

The Artificial Intelligence Reviews (AIR) methodology was developed by the Africa Social Work and Development Network (ASWDNet) to contribute to ethical and non-colonising usage of AI. This type of review is NOT about using AI to do a literature review, but is about reviewing content generated by AI. The role of AI is to give you the content BUT not to analyse the content. Analysis remains the role of the researcher. To use this method, create a guideline or protocol that shows what you are going to search (topic and question), how you are going to search it (search words, phrase or sentence and prompts), where you are going to search it (applications, minimum 2 apps), the technical specifications of each app (the language models it uses e.g OpenAI, LaMDA etc, who developed it), how you will extract data from AI and analyse and report. Analysis of AIR includes comparison with a literature review report from any of the other methods of literature reviews reported above. Since AI has day dreams, errors and colonising content, these should be picked and reported extensively. AVOID asking AI to include sources but consider AI to be one source. This is because AI picks letters and words to create meanings from different sources but does not take meanings from its sources.

AIR process

1. Create a guideline or protocol: Identify gap, create topic and review question/s, create plans for extraction, analysis and reporting/publishing and identify a literature review report to compare with.
2. Determine search strategy: Identify search request (words, phrases, or sentences) relevant to topic; develop prompts.
3. Select search platforms: Choose the applications or databases where you will conduct your searches; specify the language models they use, such as OpenAI, LaMDA, or other.
4. Apply your search request and prompts to each application in the same manner.
5. Copy/export content: Transfer content from AI applications to programs that can hold text, such as word processors, spreadsheets, NVivo, or Covidence, in simple text or PDF format as required.
6. Data extraction: Use your research topic and question to create a criterion for extracting relevant information from the content exported.
7. Analysis: Method of analysis can be thematic, statistical, other or combination.
8. Compare your findings with a literature review report from other methods; look for patterns, trends, and gaps.
9. Identify and report issues and shortcomings: Be aware of potential errors and biases in AI content; these may include inaccuracies or “daydreams” (unrealistic outputs). Report on any colonising content and its effects on content.
10. Report your findings: Compile your analysis into a structured report or journal manuscript; include a background which encompasses theory, methodology as above, findings, analysis of errors and biases, comparison with other literature reviews, and implications and recommendations for improvement or further research.

AI applications that can generate longer answers in text

Content below was AI generated from ChatGPT on 30 September 2024 using the search request ‘AI apps where you can search and get longer text answers’ and the prompts were (1) add city, publisher, language model and year launched (2) summarise response in no more than 8 words (3) arrange according to popularity and usage. The original list had 22 apps which were screened to remove less used apps and those without year and publisher name.

• ChatGPT: Developed by OpenAI in San Francisco, CA, launched in 2022, using GPT-3.5 or GPT-4; Interactive AI for conversational responses and information.
• Bing Chat: Developed by Microsoft in Redmond, WA, launched in 2023, using a variant of OpenAI’s models; AI chat for search and information retrieval.
• Google Bard: Developed by Google in Mountain View, CA, launched in 2023, using LaMDA; Conversational AI for creative and informative content.
• Microsoft Copilot: Developed by Microsoft in Redmond, WA, launched in 2023, using OpenAI’s models; AI integration for Office applications and productivity.
• Notion AI: Developed by Notion Labs Inc. in San Francisco, CA, launched in 2022, using OpenAI’s GPT-3; Generates notes, summaries, and content in Notion.
• Chatsonic: Developed by Writesonic in San Francisco, CA, launched in 2022, using OpenAI’s GPT-3; Conversational AI for detailed responses and content.
• Perplexity AI: Developed by Perplexity AI (location unknown), launched in 2022, using OpenAI’s GPT-3; AI search assistant for detailed information.

Publishing AIR

AIR are publishable in journals, books and blogs. This type of review is acceptable for publication in the Africa Journal of Social Work and other journals listed here. Most other journals will accept this methodology if the guidelines are followed.