Drugbaron Blog

February 16, 2024 no comments

Lessons from Monte Carlo Models: Why Drug Development is Hard

“There is more than one way to skin a cat” is a rather gruesome British idiom, but its sentiment surely applies to running a successful pharmaceutical portfolio.  It is now more than a decade since Francesco De Rubertis, together with Kevin Johnson and Michele Olier, coined the term “asset-centric” investing to describe the approach to portfolio creation that still underpins the strategy at Medicxi.  And today it has earned its place in the lexicon of life science venture capital, playing a key role in generating returns of investment houses across the globe.

But it is not universal.  Indeed, many highly successful investors adopt strategies that are close to the opposite of “asset-centricity”, with large Series A rounds behind pipelines or platforms.  And even the fathers of asset-centricity themselves have said on many occasions that the approach is not suited to all kinds of assets or opportunities.

What then determines the “right” model?

The same Monte Carlo models that a decade ago helped DrugBaron refine the concept of asset-centricty can provide insight into the conditions under which an asset-centric strategy “wins”.  Doing that requires a quick review of the those models (described in much more detail here and here).

The system being modelled is shown in Figure 1.  Synthetic portfolios are created in silico, where each asset has a hidden flag as to whether it “works” or not (because, in the real world, whether an asset can be successful or not is fixed before you invest – we just don’t know which ones work until after we work them up), with X% of the assets marked as working.  A sum is then invested in each asset ($Y) and thereafter a decision is made whether to kill or continue the asset – and crucially this decision has a false positive rate (A%), where it keeps going assets whose hidden flag indicates eventual failure, as well as a false negative rate (B%) where it kills assets marked as “working”.

This cycle is then repeated multiple times, and once a fixed sum has been invested, the remaining live assets are monetised, with those that work delivering $Z and those that do not being worthless.  The total sum realised compared to the total invested estimates the return on that portfolio.  You can run the model with lots of synthetic portfolios with different conditions (X, Y, Z, A and B can be varied), comparing average and range of returns to optimise the strategy.

What this taught us that returns are most sensitive to costs, particularly during the early iterations (such that it rarely makes sense to pay extra for more information, even if it modestly improves the quality of the decision filter).  It also showed us that stringency (killing more things, even at the cost of more false negatives) is the critical parameter.

And so these became the core principles of asset-centric investing: control costs; drip feed capital; kill aggressively and only pay for directly actionable data.

But what confers these properties on the model?

Three key characteristics: the first is a relatively low initial chance of success of any of the assets (X), below around 1 in 5.  Second, the model assumes that there is a zero cost (in resource or capital) of acquiring the projects in the first place.  And lastly, the assumption that there is an inexhaustible supply of initial projects of similar likelihood of success.  In short, assumptions that describe the early-stage investing world pretty well.

But not everyone in drug development is playing this game.  The Monte Carlo models that informed the asset-centric approach only optimise the strategy for playing the game that is being modelled.

So what does the game look like to a late-stage investor or a pharma company head of R&D thinking about portfolio prioritisation? Now the pipeline is populated by assets that have a significantly increased chance of success compared to early-stage ideas (or at least they will do if you are playing the game right – the input of the late-stage model should resemble the output of the early-stage model).  In addition, it has cost a lot (often a LOT) to achieve the degree of understanding of each asset that has now been attained – meaning that assets with equivalent properties are nowhere close to being free to acquire.  And of course you may not even be able to find ONE new asset to acquire with the same likelihood of success – let alone have an inexhaustible supply.

Figure 2 shows the model for a late-stage environment.  It has almost the same structure, except now the initial assets cost something ($Q) to acquire – but in return a higher fraction of them (X) are marked as “working”.  At each cycle $Y are spent progressing the assets, and then a kill or continue decision filter is applied exactly as before; but this time with the added ability to replace the asset with a new one for $Q.  Once the specified total capital has been invested, the returns are calculated exactly as before.

What happens when we run these models is a revelation.

Now, the most important parameter is X – the fraction of good assets in the initial mix.  The cost of progressing each asset (Y), the most important determinant of returns for the early-stage model, is now essentially irrelevant.

And there’s more.  Returns are improved by LOWERING the stringency of the filter – even allowing assets to continue to be developed for a few loops after appearing to fail in case the decision was a false negative.  And paying for even a slight improvement in the quality of the decision filter (previously a bad idea) is now a sound investment.

In short, the optimized strategy for playing this new late-stage game is the polar OPPOSITE of the winning strategy for the ostensibly closely-related early-stage game.

No wonder investing in drug development is so hard! It really matters what game you are playing, and you have to know where you are on this spectrum to optimize your strategy.  But how do you know which game you are playing?  It depends on the probability of technical success (PTS) of the assets in your pipeline (the hidden parameter X).  Unfortunately, most of us are too optimistic about PTS assessments, perhaps because we struggle not to drink the cool-ade when dealing with ideas or teams we are close to.

It also explains why different investors can adopt polar opposite strategies and both still win – the key, as Medicxi have always said about asset-centric investing, is applying the right strategy to the right pool of assets.

If you behave like a pharma company on assets with too low a probability of success (perhaps because they are just too early), then you will lose money by spending too much money on assets that eventually fail.  By contrast, if you behave like an early-stage investor with a good quality pharma company pipeline, you will likely kill value by under-investing or stopping projects that have real value.

The single most important skill for investing in drug development, or managing a portfolio of assets in a company, is recognizing when the “switch point” comes for each asset.  And that is easier said than done.  There is (as far as DrugBaron knows) no useful objective heuristic to guide you – perhaps the most useful advice is qualitative: the “switch point” is the moment when the question changes from “is this something?” to “what is this?”

Not surprisingly, these Monte Carlo modelling experiments don’t tell you how to succeed in the real world (there is no cast iron money-making strategy in any industry – and if there were, DrugBaron would likely not be sharing it with you, gentle reader).  Instead, they describe how to play the particular game being modelled.  Success, therefore, depends on you correctly identifying the game you are playing in the real world.

Of course, its not as easy as it sounds – if it were, we would all be rich.  But the lesson is clear: you can’t win by copying someone else’s strategy, nor by simply blindly following your own playbook.  Instead, you need to dynamically select the right strategy according to the prevailing circumstances.  Estimating the probability of success for a specific asset is difficult, and our visibility of the external landscape (availability of assets of a given quality, for example) is always imperfect.  But these models show that returns do not scale linearly, but behave very differently (even in completely opposite directions) in different domains – your first job is to recognize which side of this “switch point” your particular program lies.

The models don’t tell you how to do anything.  But they illustrate how, if you are not thinking carefully about what you are doing just how easily you can go wrong.  Armed with the realization that a “switch point” exists you have a fighting chance of optimizing your strategy and improving your returns.  Drug development will still be hard, but your chances of emerging with more value than you started with will be materially increased.

Yearly Archive