Enabling Magicians, Not Magic ✨

Rethinking AI in Product Features

Lately I've been dwelling on the future of UX for AI enabled features - anything that today lives in the sparkle ✨ section of products.

I don't like the sparkle menu full of “AI Features,” but I'm not surprised it exists.

1. We build off of each others' UI - every dating app ended up with swiping. As far as I can tell this one came from magician.design¹, where it made a ton of sense.

2. We're trying to go fast. It's easier to add a new button to a UI than rework a product from the ground up especially when you aren't yet sure if people will adopt it or how they'll end up using it.

3. We're thinking tech first. This is grouping a set of product features by the technology that enables them, because that's the new material we got to work with. Imagine if we had an "internet" menu that we went to in products - you'd go to the internet menu for anything that required the server. You’d go there to share with collaborators, "save" your work, or refresh. That sounds so silly when we already have pull to refresh, automatic saving, and a sharing workflow!

Even within the sparkles menu, we're seeing a lot of convergence across products, with some variance in naming.

Most products that focus on AI generation of text have this set of options:

• Draft and/or create template

• Shorten / Make more concise

• Lengthen / Elaborate / Enrich

• Language and translation

• Convert format

• Simplify Word

• Modify Tone

• Summarize

So what's this going to look like in the next few years?

AI Features that move into existing UI

I cannot say this often enough. Sparkles are exciting because it's new, but existing workflows are what will be sticky (like the autocomplete features in Github Copilot). It's easy to see how some of the existing functionality easily maps to an existing UI element that can be better:

• Convert Format - Today in a text editor, you can highlight a set of paragraphs to turn them into bullets. One could argue you this is the ideal feature for a new toolbar button², but there's no reason not to highlight one paragraph of text, hit the same button and end up with your content as bullet points.

• Make more concise - This feels like a “better” version of grammar check, and could likely have a similar UI.

• Language and Translation - These menus also already exist in many text products. It’s usually buried today (file → option → language). While converting languages is a fun AI demo, it doesn’t need the prominence it has today.

Another thing to note is that these features don’t need to leave in only one place.

Better draft tools

Another hallmark of these is the "Draft" (generate) functionality. This could also be called the “ask” functionality in the ask and adjust framework. If we were to think about just existing UI, “draft” looks the most like “start from a template.”

But, creating a custom draft is much more complex than an off the shelf template. Part of what worked well about magician.design was there was already a set of constraints around what the AI needed to generate. “I am making an icon within this Figma file”, or even “I am writing UI copy” is a more narrow use case generic text writing.

It’d be magic if you could just say “write my newsletter” and it appeared, but we’re definitely not there. And most people still haven't learned the most effective way to ask for the type of output they’re looking for. Starting from a blank page is hard. Right now we expect users to do it through trial and error or by copying content from experts (who just did the trial and error process themselves).

And frankly, no matter how good the technology gets, users will still need to know what to externalize to get results. That’s what we need new UI for.

As these products get more sophisticated, we’ll start to see new experiences for the “draft” stage of content that provide much more UI and structure than what we have today.

Suggested Customization

One option is figuring out what prompts result in better experiences for a given product. Then, asking the user to get that information upfront, like the custom instructions for ChatGPT, so they aren’t doing prompt engineering on their own. This might end up looking more like the long onboarding surveys that were popular in ~2015, but I hope there’s a better option. This could be for a user (as it is in this example below) or for a specific piece of work.

Incremental Generation

Another option is providing incremental steps in generation to allow the user a chance to refine before seeing a “final” product.

One example of this is Tome' taking a two step approach. The user defines a presentation topic, modifies an outline of the presentation, and only then moves into slide design.

By seeing the outline, the user will remember things that felt important they might not have written down immediately.

Tome’s presentation outline functionality

Better Editing Workflows

One of the big advantages to having generative AI within a document editor (vs. going back and forth to ChatGPT) is being able to adjust³ the work over time instead of copy and pasting back and forth. Some of the new “AI Native” products do this already.

More (!) Incremental Generation

I think the most likely outcome here is that we end up with more incremental generation, like when we draft.

With current tooling, it’s hard to get a deterministic outcome or be able to say “keep X, change Y” - even for something as “keep the first sentence but revise the second.” I anticipate that we'll be able to highlight specific sections of text to request targeted revisions and adjustments, as opposed to regenerating a full section.⁴

This menu could be much more powerful.

In fact, some of the newer AI native tools already have UI like this.

This type of UI from fixkey.ai, but with a focus on editing vs. things that can be buttons.

Human-like Editing

We’re currently in an uncanny valley where we have “✨ menu AI” or “full chatbot UI.” I’ve already made the case that a lot of things in the menu (like turn into bullets) can easily fit into our existing UI paradigms.

But some things are more of a back and forth. We’re all very familiar with having a person edit our writing - and prompting them for where we’re at. “This is an early draft of an idea” receives different feedback than “I want to publish this soon, but I need copy editing.”

I think we’ll expect AI to match these experiences. Some of them can build on what we have today, like comments and the “suggest changes” workflows.

In this case, I think we’ll see AI pop up in a more human-like fashion vs. being an ongoing chatbot next to our document⁵. Other functionality may be re-used here. For instance, a comment on a draft might suggest that a section would have more impact as a bulleted list vs. prose.

These comments may also have benefits over having a peer review writing - people often have vague feedback (“I don’t love this section”) which can be difficult to resolve and take a lot of asynchronous back and forth. The AI would be more responsive in those cases. Commenting UI will likely need to be expanded to allow discussion or automatic remediation.

I think it remains to me seen whether this is an “on demand” as a review at a given step, or an “always on” process⁶.

A better version of what we have today

As we see what works best for people, these features will get interwoven throughout our tools.

Once we’re there, it won’t feel like our AI wrote something. It will feel like we wrote something, and that we’re a much better writer than we were just a few years ago.

And once that happens, this moment will stick out as the funny point in time when we had the ✨ menu - a hallmark of 2023/2024. Just like how all the 2007-era phones stand out in Gossip Girl Season 1.

My personal favorite was the Sidekick, but the Atlantic compiled this nice set of the wide range of devices Gossip Girl featured.s

1. Where'd the Sparkles come from, anyway? You've probably noticed that pretty much every product has landed on "sparkles" as the indicator of an AI product/features. I'm not the only one who has noticed this (here's a thread with a lot of people postulating).

As far as I can tell, Sparkle originally came from Diagram’s Magician announcement, shortly before ChatGPT came out in November of 2022.

Magician was interesting because it was the step beyond "automator" - which allowed users to automate basic workflows. Magician felt more like "create something out of nothing" which fit with the 🪄 magic wand branding, especially in the context of being within a larger creative suite.

For this product, this iconography makes a ton of sense - I've said before that developers want to be magicians (not have magic) and my experience is that designers feel the same way.

2. Yes, I worked at Microsoft and I am drawn to “add a new section of buttons to the ribbon!” (like this) but here it really makes sense.

3. I link to the piece on Ask & Adjust workflows a lot because it was the first thing to neatly distinguish between a draft and then collaborating with AI to get to a final work.

4. As a small prototype of this, I asked ChatGPT to re-write this entire section with some feedback, and the outcome wasn’t good. When I provided the first part as “context” and asked for a re-write of just the last sentence, the outcome was much better.

5. This doesn’t mean the AI needs to pretend to be a person. It’s that the AI is meeting a person where they are and how they’d collaborate with another person. There’s a big difference between Clippy style language - “I see you’re trying to X!” and “Make tone more formal?”

6. With today’s technology there would definitely be a high financial cost to having this be “always on,” so I think we’re more apt to see a “request review” workflow in the short term.