Outropy <> Slack AI

tl;dr: Slack AI introduces basic summarization features, but lacks the context-awareness and user-centric design that set Outropy apart. We are now excited to bring Outropy beyond Slack, with integrations for GitHub, Jira, Figma, and Google Calendar. You can join the private beta now.

As engineering leaders with decades of experience in tech, Rodrigo and I founded Outropy to combat the overwhelming chaos of the modern enterprise. As we talked to our potential users, one tool in particular was flagged as the biggest source of entropy: Slack.

While Slack’s flexibility fueled its success, its lack of structure requires users to read through lots of messages spread across many channels to understand what is happening across the organization. As one of our users described, “finding what’s going on in Slack is like calculating your bank balance by manually summing each transaction.”

The issue is well-known. In March 2023, five months before Outropy’s incorporation, Salesforce announced that it was in beta with its AI support in Slack to tackle it:

We knew it would be challenging to compete directly against Salesforce within Slack. However, this integration has always been just the first step in building the AI Agent platform for modern enterprises. As long as we made our product available to users before Salesforce did, we would validate our assumptions, learn from real-world users, and show our technology in practice.

After two intense months of coding and private beta testing, Outropy launched publicly in November 2023:

Between November ‘23 and January ‘24, we had around 11,000 users from 40 organizations in Outropy beta. We processed over 100,000 messages from 9,000 Slack channels. Observing people using Outropy’s Slack integration taught us a lot about what they value, and we used this insight to build our roadmap for the year.

Slack AI joined the channel

During our beta period, Salesforce’s upcoming product naturally remained a hot topic in conversations with users and potential investors. That’s why we were thrilled when Salesforce announced the general release of Slack AI last week.

Slack AI is available as a paid add-on to any organization on a paid plan. It costs $10 per user per month, on top of what you already pay for Slack (and before taxes). In our team’s case, we are paying $18.75 per user per month—$8.75/user from the Slack Business plan plus $10/user for the AI add-on.

It has three features: summary channel or threads, daily recap, and AI-augmented search. The summaries and recap features most closely resemble Outropy’s offerings, so we will compare those in this article.

Channel Summary

The simplest use case for summarization is when you have a wall of messages on a Slack channel or thread and want to know what is happening. You can get the summary for a channel from Slack AI by pressing the button on the top and selecting the time frame you want to summarize.

I believe Slack sends the messages from the period you selected to the LLM and asks it to summarize them with minimal prompt engineering. This architecture is sometimes called a thin wrapper, as you can get very similar answers by manually copying the messages into ChatGPT.

Unfortunately, relying too much on the LLM results in this play-by-play type of output we see in Slack AI’s channel summary. It doesn’t feel natural and has a terrible noise-to-signal ratio.

One particularly annoying thing about this approach is how it summarizes your own messages. Here’s an example of me summarizing a channel in the Outropy team’s Slack:

We used a very similar design for our prototype for Outropy, deployed to a few Slack workspaces from design partners between June 2023 and late August. The issues discussed above became clear very soon, and we quickly changed the implementation to a more powerful RAG (Retrieval-augmented Generation) data pipeline. We used vector search to find content that could provide the LLM with contextual information to formulate better summaries. After confirming that discussions were rarely self-contained within the messages sent to the LLM, we added contextual information from other channels, time periods, and reference data.

We also added follow-up actions. Those are time-saving automations automatically suggested by the AI in our first use of OpenAI function calling.

By the end of Summer Outropy’s channel summary looked like this:

Despite these improvements, we noticed that the channel summary feature still wasn’t being used as often as we had hoped. As we investigated further, we realized that offering time-bound summaries for channels had little value to users. What they wanted was to right-click a message anywhere in the chat and be presented with helpful context around it.

We built click-to-context, which quickly became one of the most used features of our beta.

By the end of our beta period, 80% of active users used Click-to-Context at least once a week, whereas less than 10% used channel summaries.

Daily Recap

Something interesting in these conversations with users on the usefulness of channel summaries was confirming that managers and senior engineers do not want to know about every discussion that happened—not even if summarized by the smartest AI in the world.

Their ideal experience would be to get an overview of what is happening across projects and topics. If they need to know more, they’d double-click and ask for a summary of a specific discussion. And if they need even more detail, they’d double-click on the summary and read through the raw messages.

However, they were also clear that a deal breaker for any tool would be FOMO, Fear of Missing Out. Users said they would immediately stop using a tool if it occasionally skips important discussions, requiring them to double-check its work.

The Slack AI team probably heard similar input from their users and made its Daily Recap feature one of its highlights.

The first thing I noticed testing this feature is that when they say daily, they mean it literally. I installed Slack AI in our lab in the afternoon, but I had to wait until the next day to see it in action.

More than just an annoyance, this is a serious issue for this kind of tool. A morning report_ _sounds good until something important happens between 7 a.m. and when you start working. Nothing is worse than coming to a 1:1 with your boss at 9 a.m. and being greeted with “What’s up with the outage?” when your daily recap didn’t mention the outage that started at 7:47 a.m.

But even if we could find the perfect time, having it once a day is not enough. Busy people spend hours in meetings and often only have a few precious minutes between meetings to catch up with what is happening—frequently while walking from one meeting room to another. That’s the perfect time for such a recap.

When building Outropy’s recap-like feature, many of our design partners and users had teams across different time zones. This distribution meant that a fixed time for Outropy’s personal briefings was not an option, so we delivered the report whenever the user asked for it by typing /catchmeup.

After correlating our analytics, we noticed that we could predict when they asked for a report: whenever they came online, be that first thing in the morning or when checking their phone while walking from one meeting room to another.

Slack doesn’t have an API to notify clients when a user goes online/offline, so we had to build our own presence system using a mix of signals and hacking around some nasty rate limiting in Slack’s API.

But, back to Slack AI, I found it surprising that you must manually select which channels you want in your recap.

As you might imagine, this is an invitation for FOMO. Did you create a new channel but forget to add it to the list? It won’t be in your summary. Someone is asking about your project in a channel you didn’t even know existed? Tough luck.

Luckily, I had seen this the day before. With all channels selected, the next morning, I finally got my recap:

I was surprised that the daily recap was just a list of summaries for each channel. The content is the same as individual channel summaries but presented as one long screen instead of the side panel.

Aside from the issues discussed previously, this misses the point of a daily summary. On a typical workday, one single topic will be discussed across many channels by different subsets of people. A recap’s biggest benefit is a centralized, cohesive summary of these conversations scattered across multiple channels.

It also mutes channels added to the summary by default, which was weird to me. Wouldn’t the channels I want in the recap be the ones I care about the most?”

I understood this better after reading their design team’s blog post on the launch. The Slack AI team’s focus is on making users use channels more efficiently:

I see how this makes sense for the company that sells Slack, but our experience shows that conversations naturally happen everywhere, including in channels dedicated to something else. An engineer will talk about a JavaScript bug on the #design-team channel, a manager will ask the best day for the offsite on the #pytorch channel, and someone will announce there’s cake in the kitchen in #okrs-2025.

One of the core principles for Outropy is that work happens everywhere, and our role is to ensure users don’t have to waste their precious time scanning all the different places for information they need to do their jobs. To this effect, we designed Outropy around the concepts of Conversations and Discussions.

Conversations are longer-term topics being explored (e.g., Are we ready for the release next week?). They are continuously updated through discussions, which are time-bound interactions inside channels. A single discussion can affect multiple conversations, and it is crucial to keep track of them and understand their impact.

Following this paradigm, our personal briefings focus on updates to conversations:

The briefing is centered around these conversations. As many conversations are active at a given time, we bring only three that Outropy knows are the most relevant to you, considering what we know about your interests, priorities, and upcoming agenda.

We also tailor the summary and action items to you and your role. For example, a salesperson looking at a system outage summary will see one that focuses on customer and revenue impacts, while an engineer will see more technical details.

Here is how we show updates to a conversation in the report:

To build this user experience, we had to evolve our data pipeline beyond the simple ‘just send the messages to the LLM’ approach used in Outropy’s prototype. We will publish technical details about our approach soon, but in simple terms, we group messages into discussions and then process those to infer that conversations exist and how they might change over time.

Our contextual information also evolved from a simple preamble in our prompts to a probabilistic graph that maps how we saw people working and identifies non-obvious relationships between people and objects.

Iterating Forward

Releasing something like Slack AI in just over twelve months is an impressive feat for a massive company like Salesforce, with a million competing priorities and millions of users to cater to. However, the current state of Slack AI feels more like an early 2023 demo than a finished product sold at a steep monthly fee of $10/user.

I have no doubts that the Slack AI team will address the various blind spots and limitations as they learn more from users. Still, our experience at Outropy makes me believe that the product is fundamentally limited in ways no UX improvements can overcome.

The challenge is that although Slack is a critical tool for collaboration in modern organizations, it is not the only place where work happens.

For the particular user persona we target with Outropy, engineering leaders, and managers, the most valuable work occurs outside of Slack. It happens when they are building a project plan on Google Docs, reviewing code on Github, prioritizing the team’s backlog on Jira, or trying to organize their week on Google Calendar.

While we can ingest data from these platforms, we realized that to support Outropy’s vision, we have to expand beyond Slack. We must be where our users are, and aside from IDEs such as VSCode and Slack itself, this means the browser.

That’s why, in February, we started working on Outropy Companion. It brings the next generation of our model anywhere you need—just press ⌃+⌘+o and it pops up.

Here is how it looks like at the moment:

Outropy Companion will be generally available in late Spring, but we have an ongoing private beta, and you can sign up for it here.