Preview Mode Links will not work in preview mode

ClickAI Radio


Dec 15, 2022

In this episode, I talk with the CEO and founder of an organization that has been applying AI to help them develop products. Will AI help you develop your products faster? Come and see.


Grant
Hey, everybody, welcome to another episode of ClickAI Radio. So today I have this opportunity to speak with one of those brains out there in the market that's being disruptive, right? They're making changes in the industry in terms of not only the problems are solving, but it's the way in which they're solving the problems using AI very fascinating. Anyway, everyone, please welcome Paul Ortchanian here to the show.

Paul
Hi, nice. Nice, nice of you, happy to be here on the show. 

Grant
Absolutely. It's very good to have you here today. When I was first introduced to you. And I started to review your material what it is that your organization has put together as fascinated with the approach because I have a product development background and in in the software world. AI was late comer to that right meaning over generations when I saw the approach that you're taking to that I'm interested to dig more into that. But before we do that big reveal, could you maybe step back and talk about the beginning your journey? What got you on this route? And this map, both in terms of product development, and technology and AI itself?

Paul
Yeah, absolutely. So I started out as an engineer, headed down to San Francisco in the early 2000s. And, and I was more of a thinker than an actual engineer, or just be the type of guy who would figure things out by themselves. But if you were to ask me to really do things that the real things engineers do, you know, creativity was there, but not the solutioning. So being in San Francisco was a humbling experience, I guess, Silicon Valley, you get to see some really, really good engineers. So I had to make a shift in my career. And since I had a passion for user experience, the business aspect, product management was a great fit a function I didn't really understand. And I got to learn and respect, and did that for about 10 years. 

In the mid 2000s, and 10s, I basically moved back to Montreal for family reasons and cost of living, of course in San Francisco. And I started a company called Bank Biddick, which in French stands for public bath. And the idea is that most what I realized in Canada was that people here in accelerators, incubators and, and startups just didn't understand what product management was. So they didn't really understand what they do and how they do it. And I saw a lot of organizations being led by the marketing teams, or the sales team and being very service oriented and not really product LED. 

So basically, it basically stands for public bath, which means every quarter, you want to basically apply some hygiene to your roadmap, you have a galaxy of ideas, why not go out there and just, you know, take the good ones and remove the old ones and get rid of the dirt. And we started with that premise. And we put we said, well, what does a product manager do on a on a quarterly basis? Because a lot of the material you'll read out there really talks about, you know what product managers should do in terms of personas and understanding the customer's data and this and that, but nobody really tells you which order you should do it. Right. If that was my initial struggle as a product manager, do you try to do it all in the same day and then you realize that there's not enough time? So the question is like in a one quarter 12 week cycle, as my first three weeks should be about understanding the market shifts the industry, the product competitors and and the users and then maybe in the next three weeks working with leadership on making sure that there is no pivots in the organization or there are some some major strategic changes and then going into analyzing the DIS parking lot of ideas and figuring out which ones are short term and re and making business cases in order to present them for, for the company to make a decision on What to do next on the roadmap. 

So there is a process and we just call that process SOAP, which goes in line with our public bath theme. So the idea was like, let's let's give product managers SOAP to basically wash their roadmap on a quarterly basis. And, and that's what being public does. And we work with over 40 organizations today so far, on really implementing this product LEDs process within their organizations, we work with their leaders on identifying a product manager within the organization and making sure that marketing support sales, the CFO CEO really understand how to engage with them what to expect from them, and how product manager can add value to to the organization. And so they just doesn't become, you know, this grace towards them as many features as you can pump out, right.

Grant
Oh, boy, yeah. Which, which is constant problem. The other thing that I've noticed, and I'm wondering if, and I'm sure that your SOAP methodology addresses this, it's the problem of shifting an organization in terams of their funding model, right? They'll come from sort of these project centric or service centric funding styles, and then you've got to help them through that shift to a different funding model round products. You guys address that as well.

Paul
Yeah, we address that a lot. One of the things we always tell them is if you are a service professional services firm, and you know, I have no issues basically calling them that. If and I asked them like do you quantify staff utilization in percentages, like 70% of our engineers are being billed? Right? Do we basically look at the sales team? How many new deals do they have in terms of pipeline? Are we looking at on time delivery across those, so double use that to serve the sales team closed? And what is our time and technical staff attrition, that usually tends to be identifiers of you being a service firm? And we often ask them, well, let's let's make the shift, when we identify one little initiative that you have that you want to productize because they all these service firms, really all they want is recurring revenue, then the service is tough, right? 

That you constantly have to bring in new clients. So this recurring revenue, the path to recurring revenue is, you know, being able to say, Okay, I'm going to take two engineers, one sales person, one marketing person, one support person, and a product manager. And those guys collectively will cost me a million dollars a year, and I'm going to expect them to basically bring me $3 million in recurring revenue. That means that they're, they're no longer going to be evaluated on staff utilization, they're no longer going to be evaluating the number of deals they're bringing in. And they're, they're really going to be evaluated on how are they releasing features? Are they creating value for those features? are we increasing the number of paid customers? And are we basically, you know, staying abreast in terms of competitors and market industry changes. 

And so that's a complete paradigm shift. And that transition takes a while. But the first seed is really being able to say, can you create an entity within your organization where the CFO accepts that those engineers are dedicated and no longer being, you know, reviewed in terms of their utilization rate in terms of their know how much they're billing to customers? Once they do that shift in the recipe is pretty easy to do.

Grant
Yeah. So it's become easy. So the thing to I've seen and experienced with, with product and product development is the relationship of innovation to product development. And so I see some groups will take innovation, and they'll move that as some separate activity or function in the organization, whereas others will have that innate within the product team itself. What have you found effective? And does self addressed that?

Paul
Yeah, I mean, we always ask them the question of what how are you going to defend yourself against the competition with the VCs that have to call their moat, right? And that defensibility could be innovation, it could also be your global footprint, or, you know, it could be how you operationalize your supply chain make things really, really cheap, right? Every company can have a different strategy. And we really ask them from the get go. We call this playing the strategy, we'll give them like eight potential ways a company can, you know, find strategies to differentiate themselves? And the first one is first the market? 

And the question is, it's not about you being first to market today. But do you want to outpace your curlier closest rivals on a regular basis? And if so, you know, you need an r&d team and innovation team who is basically going to be pumping out commercializable features or r&d work. And then we always give him the two examples, the example of Dolby Dolby being completely analog in the 70s, but really banking on their r&d team to bring him to the digital age and from the digital age to set top boxes to Hollywood and now into Netflix compression, right? 

So they basically put their R&D team as the leader to basically keep them a step ahead of their competition. But it but on the other hand, we also Welcome, you know, talk about Tesla, where Tesla is basically doing the same thing, but they're not doing it for intellectual property like Dolby, they're not suing anybody are actually open sourcing it. But there's a reason behind it where that open sourcing allows them to basically create the, you know, what we call the Betamax VHS issue, which is making sure that there's compatibility across car manufacturers for Tesla parts and overproduction of parts that are Tesla just to increase their supply chain, right? So we ask them, Do you want to be that company, if you don't want to be that company, then there's other ways for you to basically create defensibility, it could be regulatory compliance, if your industry requires it, you can go global, you can go cross industry, you can basically create customer logins, how just how SAP and Salesforce love to basically just integrate workflows with like boots on the ground, professional services certified teams, right? 

And or you can basically review your process and make sure just like Amazon, that you're creating robots to do human work in order to just basically do it cheaper than anybody else. So there's ways of doing it. And I would say that if you were in AI space, especially, you know, it's important to make sure that, you know, are you really trying to innovate through AI, because you can get a lot of researchers doing a lot of things, but that's not really going to help you create commercializable ideas. So from the get go, the leadership team needs to, you know, at least make a hedge a bet on, you know, expansion, innovation, or creating efficiencies and just, you know, decide and let the product management team know in which direction they're gonna go planning on going for the next six years. Please.

Grant
I love your last comment there, Paul about about getting the leadership team involved. It seems that many times in organizations, this challenge of making the change sticky, right, making it last making it resonate, where people truly change their operating model, right, they're going to start operating in a different way, their roles and responsibilities change, what is the order in which things get done all of those change, when they start moving both into this AI space, but you know, product driven just by itself, even without AI has its own set of challenges? So here's the question I have for you. As you move companies through this transformation, that's part of your business, right? You are transforming the way companies operate and bring about better outcomes. How do you make those changes sticky? Because this is a cultural change? What is it you guys have found it's effective?

Paul
Or it goes back to our name public bath and SOAP, right? Because the idea is, you take a bath on a regular basis hygiene is something you do regularly, right? So we ask these organization, if we give you a process where you know exactly what the product management team is going to do with you with the leadership team in order to prioritize your next upcoming features, then can you do it in a cyclical way, every quarter, you need the product manager do the exact same process of revisiting the competitors, the industry, the market, as well as like the problems that you have with your premature customers, bringing it back to the organization, asking if the strategy is still about expansion, innovation, efficiencies, identifying new ideas, clearing up the parking lot of bad ideas, etc, and eventually making the business case for the new features in order for them to make a commitment. So if we do this in a cyclical way, then the product role becomes the role of what I'd like to call the CRO, which is the chief repeating officer, because all the product manager is doing is repeating that strategy and questioning the CEO, are we still on? Are we pivoting or if we pivot? 

What does that mean? And if you're doing it on a three month basis, what that allows your company to do is to make sure that the marketing and sales and support team are going along with what the engineering team is going to be delivering. So this is what I usually see most product organization where a decision has been made that the engineers are going to be building a particular feature, the sales and marketing team just waits for the engineers to be Code Complete. And once a code completes, done, they're like, Okay, now we're gonna promote it. But my question is that it's too late. Right? You really need so I always show the talk about Apple, how Apple would basically go out in front of millions of people and just say, here's the new iPhone 13. And we came up with a new version of Safari, and we're updating our iOS and we're doing a 40 Other changes. And the next thing you want considered an Apple store and you know, everything has changed. The marketing has changed the guys that the doing the conferences, and the lectures and the training are all talking about the new supplier, the new iPhone, and you ask yourself, How did how did Apple know and to organize the marketing support and sales team in that in such a way that the day that the announcement has been done? Everything is changed. So that means that it's not just the engineering team's responsibility to get to Code Complete. 

It is a collective responsibility where marketing support and sales are also preparing for the upcoming releases. And and the only way you can get that type of alignment is If every three months these these parties, technology, product, CEO, CFO, sales, marketing and support can get together and make a clear decision on what they're going to do, and be honest enough of what they're not going to do, and then work collectively together on making sure that that those are being delivered and prepared in terms of the size of the promotion that we're going to do, and how are we going to outreach how's the sales collateral going to change? How is the support team going to support these upcoming features. And so everybody has work to do in that three months timeframes. So and then that if we can get to that cyclical elements, I think most companies can create momentum. And once that momentum has is generating small increments of value to the customers, then you base start start building, what I like to call reputational capital, with the clients, with the customers with the prospects. And eventually anything you release the love, and everything you release adds value. And eventually everybody loves everything you're doing as an organization become that, you know, big unicorn that people want to be.

Grant
Yeah, so the net of that is, I believe what you said as you operationalize it. Now there's it gets integrated into everyone's role and responsibility. It's this enterprise level cross functional alignment that gets on a campus. And the cadence is, in your case, you'd mentioned quarterly, quarterly sounds like that's been a real real gem for you. I've seen some organizations do that in shorter timeframes and some much longer. It sounds like yeah, at least quarterly is that a good nugget that you find there? 

Paul
Yeah, quarterly works, because you know, markets are set in a quarter way they operate in that way the you want results on a quarterly basis in terms of sales in terms of engagement, etc. But what's important is that which you know, a lot of engineering teams like to work agile or Kanban. And in a quarter in a 12 week timeframe, you could fit, I'd say, Let's see your Sprint's are three weeks, you could fit for sprint for three weeks variance, or you could fit six 2-week sprints. But I feel that if you were to shorten it, then the marketing team and sales teams supporting might not have enough time to prepare themselves for Code Complete, the engineers might be able to deliver but then the product manager gets overwhelmed because doing an industry research, competitor research etc. Every, say month and a half or two months just becomes overwhelming for them. Because things don't change enough in two months for them to be able to say, Oh, look, this competitor just came up with that. And now we need so so I think three months is enough time for the world to change for, you know, country to go to war for COVID to come over and just destroy everything. So pivot decisions are usually can pretty good to do on a on a quarterly basis. 

Grant
Yeah, that's good. That's, I think COVID follow that rule. Right. Hey, I have a question for you around AI. So how are you leveraging AI in the midst of all this? Can you talk about that?

Paul
Yeah, absolutely. So what we noticed is a lot of organizations who have products, so SaaS products, or any type of product, IoT products, etc, they're generating data. I mean, it's it comes hand in hand with software development. So all that data is going into these databases are and nobody knows what to do with them. And eventually, you know, they want to start creating business intelligence, and from business intelligence, AI initiatives have just come about, it's very normal to say, You know what, with all this data, if we were to train a machine learning module, we would be able to recommend the best flight price or the best time for somebody to buy a flight, because we have enough data to do it. So so we're not working with AI first organizations who are here we have, our entire product is going to be around AI, we're just trying to work with organizations that have enough data to warrant 1-2-3, or four AI initiatives and an ongoing investment into those. So the best example I like to talk about is the Google Gmail suggestive, replies, right, which is adding value to the user needs AI in the back, end a lot of data. 

But ultimately, it's not that Gmail isn't AI product, it simply has AI features in it. So and when organizations start identifying AI or machine learning, predictive elements to their product, then we go from engineering being a deterministic function, which is if we were to deliver this feature, then customers will be able to do that to a probabilistic function where Let's experiment and see what the data can give us. And if this algorithm ends up really nailing it, we will achieve this result. But if it doesn't, then do we release it? Do we not release it? 

What's the and then it gets a little bit hairy because product managers just lose themselves into it. Oftentimes, they'll release a feature and the sales team would just ask them to pull it out right away because it has not met the expectations of a customer or two. And ultimately, like what we ask product managers to do is work with leadership on really it Identifying a few key elements that are very, very important to just just baseline before you were to begin an AI project. And those are pretty simple. It's, it's really like, are you trying to create to have the machine learning module? Make a prediction? Are you or are you trying for it to make a prediction plus pass judgment? Are you trying to make it a prediction, a judgment and take action? Right? Decision automation, which is what you know, self driving cars do, will will see biker, they will make a prediction that it's a biker will make a judgment that it's indeed a biker, and we'll take action to avoid the biker, right? 

But when you when you're creating ml projects, you can easily say, you know, we're just going to keep it to prediction, right? Like this machine is going to predict something and then a human will make judgment and the human will take action. There's nothing wrong in doing that. So just setting the expectations for from the get go in terms of are we basically going to predict judge or take action? That's number one. And then the next question is whatever that we decide if it's just prediction, is that worth guessing? And who doesn't have guessed today, if it's a human? Is that how accurate is that human? Let's quantify. So this way we can compare it against what this machine is going to do? What is the value the company gets out of that gas being the right gas? And what's the cost of getting it wrong? So oftentimes, we forget that humans to get it wrong to and if humans get it wrong, there are huge consequences to organizations that will overlook but as soon as machine learning does the same thing, we're ready to just cancel hundreds of $1,000 of investment. 

Grant
Yeah, that's right. Yeah, we tossed it out. So the use case, I'm assuming would be you would leverage AI to say enhance a product managers abilities to either predict outcomes of some product development activities, or releases or things like that, would that be a kind of use case where he looked apply?

Paul
Well, not a product managers, I would say the product manager, we'd look at it software, let's take the software of a website that tries to predict your if people qualify for a mortgage loan, for example, right? So you have enough data at that point to be able to automate, what's the underwriting process that humans do of validating whether or not somebody's eligible for loan? Well, we could take all that data and just make a prediction of that person's fit for a particular loan. Now, if we were to say, well, it's just going to be the prediction, but we're not going to give this person the loan, we're still going to ask a human being to pass judgment that that prediction was the correct one, and then take action to give or not give him a loan. 

So let's say that's the machine learning module, we're going to add to our to our feature. Now, the question is how this underwriting department in the past 10 years, how often did they really screw up that, you know, and issued loans to people that were that couldn't pay their loan, right? And realize it's 40%? Were like, Wow, 40%? Could this machine learning be as accurate as damn plus one, right? And, and then we ended up realizing that yeah, this, whatever we delivered is 33% accurate, and not 40% plus one accurate now is it still worth putting out there we spent $100,000 into it, and then you know, then it's up to the product manager to basically be able to put this thing in place and say, but look, you know, underwriting is a nine to five job currently in our business, and it cost us this much money. 

On the other hand, if there's this machine learning is 33% accurate, but it's actually doing it 24/7 365 days a year, and it's only going to improve from 33 to 40. And if it goes above 40, then we the savings for our organization are this much money. So it is really the product managers job to be able to not only talking about the business KPIs, but also the what the AI machine learning KPIs we need to achieve and what the impact of that would be if we get it right. And I think that the biggest issue we have as product managers in the AI space is if we were to go and do this all there everything that we need to create AI, like the day data ops, selecting the data, sourcing it, synthesizing it, cleaning it, etc. The model ops, which, you know, comes down to multiple algorithms, training those algorithms, evaluating tuning them, and then the operationalization. If you do all these steps, and you get to 80 to 20% accuracy, and your target is at 70% accuracy, right? What do you do with it? 

Because you had to do all this work anyways, it cost you tons of money and time. And so how do we get the leadership team to say this AI initiative has enough value for us that we're willing to live with the consequences of it getting it wrong, or we're willing to actually have it supported by human for the next six months to a year until we basically trains itself and gets better? So it's how do you get this openness from from from a leadership team? Because what I've often find delivering AI projects is every time you deliver an AI project, and it's misunderstood in terms of its output, and everybody thinks it has to be 100% accurate, the second and goes wrong. It's the political drama that you have to go through in order to keep it alive. is just it's just overwhelming, right? So miners will set those expectations up front and tool, the product managers with the right arguments to make sure that they the expectations are set correctly.

Grant
Have you ever worked with or heard of the company called digital.ai? Are your familiar with them? digital.ai, maybe not. Anyway, they have been working in a similar space as you but not so much of the product management level. What they're doing, though, is they're, they're looking to apply AI to the whole delivery function. So so you can you see, the product manager is above this, and is making sort of these KPIs and other estimate activities and the planning out. But then there are all these functions under there that of course, do the delivery of the product. And so they're working on the tooling spectrum, I think they acquired I think, was five different companies like in the last nine months, that they're integrating these and then building this AI seam or layer across that data across delivery with that purpose and intent to do that predictive not not only backwards analysis activities around AI, but predictive, which is what's the probabilities, I might run into the problem, or some problem with this particular release, right, of this product, right, that we're about to send out, now might be an interesting group for you to get connected with.

Paul
Yeah, I know, it's funny, because we're there. There's a local company here in Montreal that does the same thing. It's really about like data scientists are really expensive, and they're really hard to find, and there's a shortage of them. So, you know, the lot of organizations are trying to find like a self serve AI solution where you can build your AI using their AI. But ultimately, what they're doing is taking your data and delivering 123 or 10 versions of the machine learning module, it's up to you basically, judge which one is going to work the best for you, but they actually operationalize it, put it out there for you, and really automate the whole thing. So this way, you're not dependent on humans, I love that I really love that I think your organization should have one of those. But that still means that there's a dependency from the for the product manager to know that it's, it's data, like end to end, be able to clean it be able to tag it and then feed it to the to these machines, right? And I think that part is also misunderstood. Because Do we have enough data? Is there bias in the data and all that needs to be understood and figure it out? Because, you know, you could say like, Hey, we put it to this big machine. And we ended up with a 20% accuracy on the best ml that it out, put it, but that's still not good enough? Because we're trying, we're aiming for 87? And what does it mean? What do we need to do to basically get it to 87? We're gonna have to review the data bringing some third party data, you know, and it's, and that's, that costs a lot as well. So, yeah,

Grant
Do you think AutoML solutions play a role here like, Aible, I don't know if you're familiar with that platform, you know, that the goal is to try to reduce the amount of dependency that's needed on the data science. Scientists themselves, right. And but it's, it's still doesn't remove all of the data cleansing part, but it does help take care of some of the certainly the low level data science requirements, you think you think that's a viable solution in this area? 

Paul
I think it is. I mean, it's, you know, we went from rule based AI, where data scientists had to do good old fashioned AI, which was a feature engineering, right? Putting the rules themselves to machine learning AI, where, you know, we had to train the data that we needed, were so dependent on these data scientists. And now we're getting to v3, where we have these tools. And you know, there's a data dependency, but there, they also don't have such a high dependency on data scientists are and you know, figuring our algorithms and etc, we could just basically have these prepackaged algorithms that could basically output us any types of solution. What I tend to like, I've seen this a lot in a lot of companies. There's some companies that are very, very industry specific, right? So they're providing AI for E-commerce to be able to provide better search with predictive elements based on the person's browsing history. I mean, I, I'm not sure, but the ones that are providing every ML imaginable, so you could use it for supply chain, or you could use it for something else. I know it's dependent on data. But again, these algorithms, you can't have all the algorithms for all scenarios. 

Even if it's supply chain, some person has perishables and there's ordering bananas and the other person is ordering, I don't know water coolers, and those, those don't have the same rules, right. You know, so it's, it's important to just, I think that maybe in the coming years, we'll have a lot of companies that are really going cross industry, just like we're in E-commerce, the other ones that are med tech, the other ones are, etcetera, the tools are the same. I mean, more or less the same, the customers are gonna get used to basically having these UI is that I'll give you your input the data in and then these emails come out, and then you choose which one and they give you probability you can retrain them and all that stuff. And I think that it's just going to get to a point where we're going to have these product managers who are now responsible of kind of training the Machine Learning Module themselves, you know if it's going to be the product manager, or if it's going to be some other function, where I think it does definitely fit inside the product managers?

Grant
Well I do is, I think it's because they still need to have what we would call the domain knowledge and in this domain of building products, yeah, AI, at least at least in this phase of the life of AI, where we are today for the foreseeable future. I think the product manager needs to be involved with that. Sure. So.

Paul
It comes down to intuition, right, somebody has to have like to build that intuition about what a model is relying on when making a judgment. And I think that, you know, with product managers, the closest one really, maybe in bigger organizations, it's the person who's managing analytics and data, but in smaller startup organization, I can definitely see the product manager putting that 

Grant
Yeah, absolutely. Paul, I really appreciate you taking the time. Here today on this been fascinating conversation. Any last comments you want to share?

Paul
We have tons of articles that talk about so we're very open source as an organization. So if you want to learn more about this, we have about 70 articles on our website. Just go to BainPublic.com and just click on "Articles" and you could just, you know, self serve and basically improve as a product manager in the AI space.

Grant
Excellent, fascinating, love, love the conversation, your insight and the vision where you guys are taking this I think you're gonna continue to disrupt everyone. Thanks for joining another episode of ClickAI Radio and until next time, check out BainPublic.com.


Thank you for joining Grant on ClickAI Radio. Don't forget to subscribe and leave feedback. And remember to download your free ebook visit ClickAIRadio.com now.