mathstodon.xyz is one of the many independent Mastodon servers you can use to participate in the fediverse.
A Mastodon instance for maths people. We have LaTeX rendering in the web interface!

Server stats:

3K
active users

Peter Smith

ChatGPT is utterly unreliable when it comes to reproducing even very simple mathematical proofs. It is like a weak C-grade student, producing scripts that look like proofs but mostly are garbled or question-begging at crucial points. Or at least, that’s been my experience when asking for (very elementary) category-theoretic proofs. Not at all surprising, given what we know about its capabilities or lack of them.

But this did surprise me (though maybe it shouldn’t have done so: I’ve not really been keeping up with discussions of the latest iteration of ChatGPT). I asked — and this was a genuine question, hoping to save time on a literature search — where in the literature I could find a proof of a certain simple result about pseudo-complements (and I wasn’t trying to trick the system, I already knew one messy proof and wanted to know where else a proof could be found, hopefully a nicer one). And this came back:

@PeterSmith Are you aware of the tendency of ChatGPT to hallucinate plausible references?

Just as a test, I checked the stated chapter and section titles in Sheaves in Geometry and Logic: they don't match the chapter and section titles in the book, at least not in the version I'm looking at. The pointer to page 391 also seems to be spurious, though I'm not really qualified to say that with total confidence.

@theohonohan Yes, that's exactly what I say in the follow up post!

@PeterSmith I think you forgot to reply to the 1st post in the second because the two posts do not appear to be linked
@theohonohan

@tfardet In fact, to be completely sure that nobody else had replied first (saying something similar) I should also have gone to the mathstodon.xyz page for the toot. I didn't do that.

(Mastodon made an egregious design decision which means that replies by third parties to boosted toots don't get forwarded to non-followers of the original tooter: github.com/mastodon/mastodon/d)

GitHubMissing replies · mastodon mastodon · Discussion #22608Hi all, I'm self-hosting my own Mastodon instance, just for me. The big problem I have is ... missing replies. This seems to manifest especially on boosted posts of people I don't follow, although ...

@PeterSmith
ChatGPT is a bullshit generator and cannot be trusted.