I started watching Westworld Season 4 yesterday, and as usual, I spent about 15 minutes catching up on what happened in the previous season. A recap: evil corporation x (Incite Inc.) creates a supercomputer (called “Rehoboam”, named after the final king of Israel), that told everyone what they would do in their lives. The supercomputer is destroyed at the end of Season 3 (yay!), and now the characters must contend with their newfound freedom.
Now, this may sound very dystopian and far-off, but in some ways we already operate this way.
Through my work in credit and loan decisioning, we see lenders spend a lot of time thinking about how models can be used to predict future loan performance. Upstart (UPST), along with many lending companies, made an entire business out of this, building best-in-class AI/ML models that would predict exactly when you would pay and how much.
Upstart touts almost 80% accuracy, but that didn’t prevent them from getting crushed in the public markets over the last two quarters (the stock is down around 94% from its high at the end of last year). Why?
Because when you tell people what they are, they believe you, and act accordingly.
Upstart told their borrowers that they were risky and deserved incredibly high rates, and those borrowers took the money, never adapting their behavior. Upstart opened up its funnel to riskier borrowers, with no tools or guidance to improve.
Similarly, Rehoboam was also accurate. In one of the first episodes of the new season, Caleb (a primary character) is working on a construction site, just as he had been in the prior season before Rehoboam was destroyed. His occupation didn’t change in any meaningful way - Rehoboam knew what he would be.
However, what did change was his capacity to fully experience his own life and choices. He became a more positive person with stronger personal relationships. Rehoboam failed as a project because, in telling us exactly what we would be, it limited our potential. Omniscience should be left to the realm of the Universe, not humans.
When I think about working with our users at Debbie, I know we must not only identify good performers, but help influence their debt payoff experience. Our models are very simple and based on time-trended improvement. The actual meat of what we do is to help people stay on track with their goals. We help them feel ok if they veer off course, and celebrate them when they meet or exceed expectations.
What we do is not easy. In fact, it’s way harder to help people improve than to predict what someone will do - we ask people to actually put in effort. So far, it’s been working - our users engage with us every two weeks and pay off $450/mo of debt on average. We tell them they have the capacity to be better, and so they rise to meet this challenge.