Teaching with AI

While definitively determining if content was generated by a human or an AI, there are trends in AI writing that can suggest a paper was written using ChatGPT or other generative AI software:

1. Language Patterns and Style: AI-generated content might exhibit certain language patterns, style, or inconsistencies that are unusual for human authors. Look for unnatural phrasing, lack of personal experiences, or overly coherent and incoherent language.

2. Inadequate Citations and References: AI-generated papers may lack proper citations or references to existing literature. They might reference obscure sources or mix up unrelated concepts.

3. Unusual References: Check for references to non-existent authors or publications that don't align with the field's established literature.

4. Inconsistent Argumentation: AI-generated content might present arguments that are not well-connected or logically consistent. It might lack depth and critical thinking.

5. Sudden Changes in Style or Tone: AI models can sometimes produce text that abruptly changes in style, tone, or level of expertise.

6. Use of Niche Jargon: AI-generated content might misuse or overuse field-specific jargon in an attempt to appear knowledgeable.

7. Use of Uncommon Topics: Sometimes AI-generated content might focus on topics that are uncommon or unrelated to the field, suggesting a lack of contextual understanding.

8. Plagiarism Detection Tools: Some AI-generated content might be derived from existing human-written papers or sources. Using plagiarism detection tools can help identify such instances.

Overall, AI-generated content often gives an impression of someone who wrote very confidently about a topic that they just didn't get. 

Here are some popular tools used to detect AI and their limitations: 

• Turnitin
  • Claims less that 1% false positive rate when documents are over 20% AI generated. 4% false positive on the sentence level. 
• Giant Language Model Test Room
  • Based on GPT 2, may not be effective for newer models. 
• Writer.com AI Detector
  • 1500 characters at a time, unknown effectiveness. 
• GPTZero
  • Unknown effectiveness: Quotes to be most reliable in a TechCrunch Article titled, "Most sites claiming to catch AI-written text fail spectacularly: Even the best of the bunch missed some." 

Note: Many detectors break down under real world conditions. There are many videos on YouTube on how to fool these detectors, and one report even suggests that tools that detected ChatGPT text with 74% accuracy dropped down to 42% when that text had been tweaked.

Since Many of the automated AI detection tools are prone to false positives, consider teaching some of the following options to help your students protect themselves in the event of a false positive. 

• Enable changelogs or version histories whenever possible: 
  • Google Docs - Click on File > Version history > See version history. This allows you to see different versions of the document and show the writing process. 
  • Microsoft Word - If you go to Review > Track Changes, Word will automatically note any edits or corrections made while writing and revising the paper. 
  • Office365 - Students are provided with a copy of Office 365. By saving their word documents to Office 365, they can access previous versions of the paper.
• Document the writing process:
  • Keep notes or annotated versions of research materials.
  • Keep outlines and rough drafts.
  • Document any tools such as Grammarly used to improve writing, and the changes it suggests.