Our learnings from building Promptly
We’ve learned a ton over the past eight months, so we thought we would publish a separate blog post documenting our learnings. Though we covered a lot of our experience in previous blogs (the importance of alignment and commitment, approaching GTM, pivoting and choosing one idea over another), we haven’t covered our many learnings while working on Promptly, and in the AI space.
Challenges Working in AI
When we started working on Promptly, we knew the space was going to be competitive. But, we couldn’t have imagined how quickly things would change in such a short amount of time. To put into context: between the time we started and ended working on Promptly, four other startups were founded with a similar value proposition (AI user interviewing platforms for product research). Even the popular survey company Typeform, began dabbling in the space through their newly founded Typeform Labs. For us to succeed, we needed to move fast to build a customer and data moat. The question that was constantly on our mind was: “How can we be 10x better?” not only from our direct competition, but also from the existing alternatives (surveys, user interviews, data analysis) for our potential customers to switch over.
Given the rate of change, it made development exceedingly difficult. AI is moving so quickly that the infrastructure and tools to support AI companies are failing to catch-up. A lot of these challenges come from the non-deterministic nature of AI (i.e not knowing for sure if GPT will correctly format JSON). Any technical decision that we’d make would become obsolete in two weeks. We constantly asked ourselves if we should continue building with old tech? Do you switch over?” We had to make a lot of difficult decisions.
Developing our AI Interviewer
While developing our AI interviewer, we very quickly learned a basic system prompt just wouldn’t cut it. It would be good 80% of the time, but go off the rails, or have hallucinations the remaining 20. A funny example of this that came out during our testing, was the AI interviewer recommending the user stop using the product entirely. Which, obviously, wouldn't pass with our users. So, we very quickly realized nothing else mattered, unless our interviewer was good. And for a solid three weeks, we worked on refining our model.
One of our biggest advancements was creating a multi-modal interviewing approach. In plain English: we essentially created a team that was tasked with running the perfect interview. While the end-user would only interact with one agent, we had multiple agents in the background who helped ensure things ran smoothly. We had a Planner who devised the interview plan based on the user’s research goal and interviewing question best-guidelines (i.e not asking leading questions, focusing on the user’s past behaviour, etc). We had the Interviewer, who led the interview based off of the interview plan, with a personality that could be changed based on the purpose of study (i.e initial market research, feature feedback, etc). We had a Supervisor who would regulate the interview, reviewing each of the interviewer’s questions before the end-user received them, and then ask the Interviewer to rephrase the questions according to the best guidelines. And finally, we had the analyzer, who aggregated the results of the interviews and packaged them into actionable insights for the product team. Through this approach, we could fine-tune every aspect of the interview experience for a specific objective and company
One of our coolest features was “Debug Mode”, which allowed an admin to review the transcripts to see how the supervisor adjusted the interviewer’s questions. (See our Planner, Interviewer, and Supervisor working in tandem)
This contributed to our technical moat, but that wouldn’t have been enough. So we spent a lot of time trying to figure out how we’d position our product in the market.
Our Product Vision
We wanted to become the ultimate user discovery platform.
From our chats with PM’s and founders, we discovered one of the largest problems PMs face is combining various data points and pieces of information, to make informed decisions on what to build next: a research management problem in addition to the research collection problem (we’re already tackling).
Our plan was to get really good at asking users the right questions to serve a PM objectives (initial discovery, churn analysis, feedback collection). To get better at question-asking, it would come with direct usage of our product, collecting data on the company, product, features, hypotheses and assumptions, user, and industry that would feed back into our model. Over time, we’d have a repository of hypotheses and assumptions that could help the company more quickly point to data when encountering a similar problem in the future (not have to run another research study). But also could be anonymized, and be used to the benefit of other companies encountering a similar problem with their products (i.e 56% companies encountering problems with their user onboarding experience have decided to do ____). The data we’d be able to collect and the inferences we’d be able to make would have made our product a lot more defensible.
Our Advice - for builders looking to dive into AI
The AI space is incredibly hot right now. Buy a .ai domain, mention AI in your pitch, and you immediately get someone’s attention. But sometimes AI can make it seem like there’s a benefit, when a benefit might not be there. In the way it is currently being applied, AI is almost like the sprinkles on a sprinkled donut. Do the sprinkles actually provide significant value to the tasting experience? No. But, do they grab your attention? 110% yes. All that being said, the question to reflect on as a builder is do you actually have problem-solution fit. AI is spicy - so most users will try it out of curiosity, but will it actually be good? How do you ensure that you create a product that will actually retain users?
A New Home for Promptly
As mentioned in our last blog post, we have decided to choose new paths for ourselves going forward. That being said, we are incredibly confident that many startups could use and integrate Promptly today. We’re looking for a new home for Promptly, if you are interested in AI-enabled User Interviewing software reach out to Harrison via LinkedIn to have a chat
