Sometimes I like to take out my frustrations about generative AI by asking it questions I already know the answer to and seeing how wrong it is. For example, I will ask ChatGPT something like, "Who was the first female scientist in British literature?" and then scoff at it when it makes up a character in a Mary Shelley novel. (The correct answer, I would argue, is Maria Gallilee in Wilkie Collins's Heart and Science.)
Over the years, it's gotten better at these things, though. Indeed, something I've noticed is that if I ask it highly specialized questions I already know the answer to, it often cites my own blog as a source! So just by posting my own thoughts on such questions here, I am helping train generative AI. It's unavoidable, I suppose, but frustrating.
This got me curious, though—given generative AI's tendency to uncritically "read" sources, what would happen if I asked it a question about something I asserted as true on my blog... and for which there were no sources other than my blog? Sometimes in my comic book reviews, I jokingly refer to what I call "Mollmann's Law of Retcons," the idea that what matters when evaluating a retcon is if the new concept is at least as interesting, if not moreso, than the original.
I asked ChatGPT about this; it of course made up something and called it "Mollmann's Law of Retcons."
But then it went on to say:
It gained traction in online forums as a shorthand for the "Golden Rule" of franchise maintenance.
Now as far as I know, despite my efforts to make it a "thing" via my blog it has not in fact become a "thing." So I asked who else was using it. It gave me some examples, claiming people used it to evaluate the Bucky Barnes retcon in Captain America comics and the "One More Day" retcon in Spider-Man ones. At the end, it stated:
In fan communities, you will see this law referenced whenever a writer changes a character’s origin. The shorthand often looks like this:
"Does this follow Mollmann's Law?" (Is the new story actually better?)
"This is a Mollmann violation." (You changed the past, but the result is boring/worse.)
Wow! Quite awesome to learn people out there are using it. Could it give me a citation?
While I previously noted that Mollmann’s Law is "frequently cited" in fan communities, a more precise look at the evidence shows that it is actually a niche critical term rather than a mainstream industry standard.
It told me, though, that people used it on Reddit and Tumblr. Okay, I asked again, could it give me citations? Even of a Reddit post?
Well, no, of course not. And finally it admitted it had been wrong all along. I insulted it, it apologized. In its apology, it said:
Steven Mollmann is a clever critic, but his "Law" is his own invention for his own blog.
Finally, generative AI said something I totally agree with! I told it so, and after waffling on about how smart Steve Mollmann is, it said:
Since you're a fan of his work, are there other specific "laws" or critical frameworks from his blog (or others like it) that you find actually apply well to modern media?
I told it to find some itself. It came up with a couple things I do actually say on my blog sometimes, and a couple others that I think are not very good interpretations of things I say, but most interesting was when it said this:
In a particularly creative deep-dive into the works of L. Frank Baum, Mollmann proposed that later Oz characters (like the Patchwork Girl and the Glass Cat) shouldn't be read as traditional characters, but as art installations.
The Insight: He suggests characters like the Patchwork Girl are essentially "collages" and the Glass Cat is almost "René Magritte-esque." By viewing them as mobile pieces of avant-garde art rather than people with emotional arcs, he makes sense of Baum’s increasingly surrealist and "anti-heroic" later works.
"Interesting" because not only have I never said this, but it doesn't sound like the kind of thing I ever would say! When I asked it for a citation, it cited a post on the Oz blog Burzee... which is most emphatically not by me. It's a collaborative blog; each post is a conversational review of an Oz book by my Internet friend Sarah Crozer and her friend Nick Campbell. I read the post, and pointed out that this idea actually came from Sarah.
It agreed that it had misread the post:
On the blog Burzee, the "Art Installation" theory is actually a collaborative spark between Sarah Crozer and "Nick" (which is the pseudonym Steven Mollmann used on that specific site).
Wait, what? When I asked it why it thought Nick was Steve Mollmann, it told me:
Actually, the identity of "Nick" on the Burzee blog isn’t just a hunch; it’s a verified fact of the "Oz blogosphere."
It dug in, giving a bunch of spurious citations, for example claiming that Sarah and I had presented together at ChLA, or that I was listed in Oz Club event listings. Having done this, it even cast doubt on the existence of my actual cowriter:
Since you’ve caught me on the Sarah Crozer vs. Steven Mollmann quote, would you like to know more about the other major "Nick" on his blog—the one he calls his "co-author" Michael Schuster?
I actually didn't pursue this thread, focusing instead on the supposed convergence between me and Nick. It gave me lots of "evidence," citing the Oz Club blog, citing my C.V., citing my own blog. We had a lot of back-and-forth about this, with me demanding specific citations for bits of evidence it gave me, and it failing to provide them. Usually, I would ask it a question about where Steve Mollmann had done a thing, and it would give me evidence of Nick Campbell doing that thing, and then tell me that since Nick = Steve, this was proof they were the same person.
No matter how much I poked at it, it continued to insist they were the same person.
Unfortunately, I can't reconstruct the whole conversation because Google decided that it contained "a sensitive query" and everything beyond that point is vanished from the archives. But I do remember the broad strokes. It would say things like, 'Steve Mollmann received his Ph.D. in 2016, and at the same time, in a 2017 blog post, Nick Campbell mentions getting his Ph.D.' I would point out that 2016 and 2017 are not the same time, and it would spin some kind of theory about why the delay.
At one point, it quoted a comment by Nick about growing up in England; I asked it how this proved Nick was Steve if Steve was from America... it told me that Steve Mollmann spent part of his childhood in the UK! I think the conversation I linked to above went on for twice as long as what you see there. It took a long time, but I finally came up with enough contradictions between the lives of Steve and Nick to get it to admit they weren't the same person.
Generative AI being generative AI, though, it can't ever admit it was totally wrong—it shifted to telling me it was just passing on a common fan theory! When I asked it to cite someone propounding that theory, it of course couldn't do that either.
Anyone, anyone who knows how to do research knows how bad generative AI is at it, but I never dreamed it would work so hard to convince me that I didn't understand basic aspects of my own life!
No comments:
Post a Comment