One of the hardest leadership decisions I’ve made happened early in my career at Sears.

The company had spent about a year building an internal machine-learning recommendation capability. It was essentially a research organization of roughly 100 people, but despite the investment, the technology hadn’t yet been meaningfully commercialized.

The CIO moved me into the organization to help evaluate what we should do with it, including ultimately whether Sears should continue investing in the internal recommendation platform.

Giving it a fair chance

Before making that judgment, I wanted to give the technology a fair chance. Rather than evaluating a research platform theoretically, I wanted to answer a more fundamental question: could we actually turn this capability into measurable business value?

So I launched a New Year cart-recommendation initiative. We used the recommendation technology to identify products that could be intelligently paired, including opportunities to pair obsolete inventory with products customers were already buying.

The product itself was very successful. Average cart size increased from roughly 0.7 to 1.4 items, and the pairing strategy helped us address roughly $3 million in obsolete inventory year over year. So we had proven something important: recommendations could create significant commercial value for Sears.

The problem it exposed

But building that product also exposed a problem. To achieve those results, we had to do a tremendous amount of manual tuning around the underlying recommendation engine.

That made me question whether the success was really evidence that our proprietary technology was strong, or whether we were creating a successful product despite limitations in the underlying platform.

That distinction mattered because the company wasn’t simply deciding whether recommendations were valuable. We were deciding whether continuing to own and invest in the underlying recommendation technology was the best use of our resources.

So I decided to test that objectively. I benchmarked our internal recommendation engine against leading external platforms, including RichRelevance and Certona, and ran traffic-split testing against our existing recommendation modules. The results were difficult to ignore. Certona delivered roughly a 20% improvement in click-through rate, while moving to the external platform would save Sears approximately $500,000 a year.

Three questions that felt like one

That put me in a very uncomfortable position. I had just demonstrated that recommendations could create substantial business value. But the same work had also demonstrated that Sears didn’t necessarily need to own the underlying technology to capture that value.

And the organizational consequences were significant. The internal recommendation organization was roughly 100 people. These weren’t people who had suddenly become bad at their jobs. The market had simply reached a point where an external platform could provide better performance at substantially better economics.

Making it even harder personally, someone I was very close to was on that team.

I had to separate three questions that emotionally felt like one. Do I believe in these people? Do I believe recommendations are strategically important? And do I believe Sears should continue owning this particular recommendation technology?

My answers were yes, yes, and no.

The decision

Ultimately, I recommended that we stop investing in the proprietary recommendation platform and move to an external solution. We selected Certona.

That decision meant the internal organization was eliminated, including the role of someone I cared about personally. That was extremely difficult, but I couldn’t allow my personal relationship, or the amount of time and money Sears had already invested, to change what the evidence was telling me was right for the company.

The move ultimately gave us better recommendation performance while saving roughly half a million dollars annually.

What it taught me

A successful product does not necessarily mean you have a competitive underlying platform. We had successfully commercialized recommendations. What we had not demonstrated was that owning the recommendation engine itself created differentiated enterprise value.

That distinction has stayed with me throughout my career, and I think it’s particularly relevant today with AI. I try to separate three things:

  • Where does technology create strategic differentiation?
  • Where can we create value by building on top of technology someone else provides?
  • And where are we continuing to build something internally primarily because we’ve already invested in it?

One of the hardest responsibilities of leadership is being willing to change the answer when the evidence changes, even when you’ve invested heavily in the previous answer and even when people you care about are affected.

A personal postscript

There is also a personal postscript to the story that has always stayed with me. The friend who was affected by the decision initially struggled after the layoff. We stayed in touch. Eventually, he joined a startup when it had only about five people. He stayed with it, the company eventually became a unicorn, and today he’s a VP there.

Seeing his trajectory reinforced another lesson for me: making the right organizational decision doesn’t require believing that the people affected aren’t talented. Sometimes very talented people are simply attached to a capability or organizational structure that no longer makes strategic sense.

My responsibility as a leader is to be rigorous about that distinction, to care deeply about people while still making the capital-allocation and technology decisions that are right for the organization.