When AI becomes the informal research assistant, reference librarian, and study partner, what happens when it invents, misquotes, and erases marginalized intellectual histories?

The term “hallucination” serves to rebrand and mystify mistakes. It turns technical errors into an opportunity to sell LLMs as more humanlike than they really are.

"When we perform risk/benefit analyses of language technology, we must keep in mind how the risks and benefits are distributed, because they do not accrue to the same people.” source: On The Dangers of the Stochastic Parrot.

Thus the risk is that people disseminate text generated by LLMs, meaning more text in the world that reinforces and propagates stereotypes and problematic associations, both to humans who encounter the text and to future LLMs trained on training sets that ingested the previous generation LLM’s output.” source: On the Dangers of the Stochastic Parrot

“Using bullshit rather than hallucination as our operative term reminds us that everything chatbots output is an attempt to look and sound right, a compelling resemblance that may or may not correspond to the true state of things. What hallucination frames as the exception, bullshit reveals as the nature of the beast.” source