I re-ran my test from the "rebuilt for attribution" post (i.e. will the tool find an answer I know exists on Space Exploration StackExchange). Overall, this version seems better. It did find what I expected it to. It then ad-libbed...badly. Judge for yourself.

Overall, much worse for my needs than typing "RPOP space" into the existing StackExchange search bar (or, even better, using Google search instead, even in these dark times).

Good

The attribution to the post is correct, the quote seems correct (I checked and did not immediately detect discrepancies), and the relevant part of the post (which was more expansive than just the question I asked "AI Assist").

Bad

• The generated summary is reminiscent of things I used to write as a schoolchild when asked to summarize something I didn't understand in my own words. The phrasing is awkward (the parenthetical in the first sentence doesn't really follow from the words that precede it) and the second sentence is arguably incorrect, though I can see why it would be put that way.

  I think it just looks really, really bad when sat right next to a summary written by the world's foremost expert on the topic (at the time, at least).

  I'd expect there to be other topics throughout the network where exactly that would happen as well: you've got a direct quote from The Person, who graciously donated their time, inside a blockquote with slightly gray text, next to a higher-contrast (flat-black?) inaccurate restatement. It's almost like something out of a satirical documentary where the laugh in the scene is getting the expert to wonder why the hell they've allowed themselves to be put in this situation next to a buffoon.

• The "key trade offs" section that follows is entirely bollocks. Honestly every time I look back at it I get angry. What weird text to invent. What an odd thing to put focus on.

• Not that I honestly expect it from what is ultimately a glorified next-token machine, but the original post has (as it should!) a link back to the primary source. Those links are omitted in the blockquote, and though the primary source is correctly mentioned in "Further Learning" it's much worse attribution than I'd given in the sourced Q+A. Hyperlinks have existed around as long as I have. We should use them.

I re-ran my test from the "rebuilt for attribution" post (i.e. will the tool find an answer I know exists on Space Exploration StackExchange). Overall, this version seems better. It did find what I expected it to. It then ad-libbed...badly. Judge for yourself.

Overall, much worse for my needs than typing "RPOP space" into the existing StackExchange search bar (or, even better, using Google search instead, even in these dark times).

Good

The attribution to the post is correct, the quote seems correct (I checked and did not immediately detect discrepancies), and the relevant part of the post (which was more expansive than just the question I asked "AI Assist").

Bad

• The generated summary is reminiscent of things I used to write as a schoolchild when asked to summarize something I didn't understand in my own words. The phrasing is awkward (the parenthetical in the first sentence doesn't really follow from the words that precede it) and the second sentence is arguably incorrect, though I can see why it would be put that way.

  I think it just looks really, really bad when sat right next to a summary written by the world's foremost expert on the topic (at the time, at least).

  I'd expect there to be other topics throughout the network where exactly that would happen as well: you've got a direct quote from The Person, who graciously donated their time, inside a blockquote with slightly gray text, next to a higher-contrast (flat-black?) inaccurate restatement. It's almost like something out of a satirical documentary where the laugh in the scene is getting the expert to wonder why the hell they've allowed themselves to be put in this situation next to a buffoon.

• The "key trade offs" section that follows is entirely bollocks. Honestly every time I look back at it I get angry. What weird text to invent. What an odd thing to put focus on.

• Not that I honestly expect it from what is ultimately a glorified next-token machine, but the original post has (as it should!) a link back to the primary source. Those links are omitted in the blockquote, and though the primary source is correctly mentioned in "Further Learning" it's much worse attribution than I'd given in the sourced Q+A. Hyperlinks have existed around as long as I have. We should use them.

• Though I understand that this experiment is deployed on StackOverflow right now, and maybe I'm misusing it by specifically asking it to look at other parts of the network (prompted by seeing a comment that said it did have training on that), but it digging up posts from Space Exploration SE and then inviting people to ask follow-up questions on StackOverflow in the last (boilerplate?) sentence seems exemplary of the State of the Thing.

If this didn't help, you can ask your question directly to the community.