Random check vs Exhaustivity

In many professions controls are carried out on a random basis.

It is a generally accepted practice that an auditor reviewing agreements or an analyst screening litterature proceeds with random checks simply because it is humanly impossible and much too costly to process each individual document.

At the same time society puts an increasing negative value to errors. A small sentence in a contract that was not reviewed or a key information missed may have very significant consequences.

At Kairntech, we have faced this exhaustivity question two times in a row in two completely different domains:

  • The life sciences industry
  • The audit

The case of the life sciences industry

Today, in the life sciences industry, any pharmaceutical company has to prove among many other things that it has not heard of any side effects on anyone in any country where its drugs are authorized for sale.

To comply, those companies have to monitor some specific databases but also the local newspapers (local literature) specialized in the pharmaceutical domain.

The good practice is controlled by National Medical Agencies in coordination in the case of Europe with supra-national organizations like the European Medicines Agency (EMA).

It is easy to understand that if a side effect is not known immediately, the consequences could be disastrous for many patients. 

Professionals must do their best to monitor local literature, social networks… in order to show their good faith but the scope is not clearly defined because in the past, it was always considered impossible to read everything.

The case of the audit practice

Today, auditors have two main closely related roles:

  • They must confirm that financial statements are sincere which means that they give an accurate picture of the reality of the company that produced them ;
  • They can also audit a company for a private customer or other third parties in the case of an acquisition. The global purpose of the auditing is to confirm, deny or negotiate the acquisition price given by the purchaser.

For the auditing activity, we have again professional organizations that have defined good practice which are often monitored by supra-national professional organizations and sometimes government organizations like in France.

It is also common practice to proceed with random checks because of the sheer number of operations you have today even in a medium company, not to mention big companies.

Controls could be defeated…

What is interesting is that both domains have been hit by scandals which shows that controls could be defeated.

  • Lack of control for Enron in the USA, Parmalat in Italy, Wirecard in Germany, Credit Lyonnais or Société Générale in France ;
  • Health fraud with the Opioid epidemic in the USA, the Servier case in France, the Thalidomide epidemic fifty years ago in Germany.

Recent progress in NLP is changing the game

The good news is that technology, and in particular Natural Language Processing (NLP) technology, is changing the game, and in particular by automating the processing of large amounts of documents (all the way from extraction to the classification and detection of words, phrases or paragraphs).

And it is not just the technology itself, but also how the technology is applied. Traditionally it has always been particularly labor intensive to annotate documents in order to create the dataset on which a trained model produces reliable results.

Annotation tooling

Recent progress in annotation tooling combines a great user experience for analysts, active learning to speed up annotations, model experimentation to allow for end-to-end and easy to create annotation pipelines, ready for industrialization.

In other words, technology and tooling are now capable of analyzing all relevant documents. 

Conclusion

It is not clear if the recent progress in technology and especially NLP is known by control authorities either professional or governmental. Mainly because it is brand new. 

Random check is still accepted for auditing while an exhaustive scrutiny of local literature or social networks is not required by FDA or EMA.

One day we can bet that a private customer will require it for its next acquisition because the consequences of bad due diligence could be very expensive. Perhaps it will reduce the one out two failures in this kind of operation.

It is unlikely that citizens will influence medicines agencies like FDA or EMA but politicians should urge them to take into account progress made by technology in order to reduce risks for public health.

Sooner or later, auditors, analysts, experts… will dedicate their time on elements requiring particular attention and by doing so the end-result becomes exhaustive, the number of potential errors dropping to almost zero.

We, at Kairntech, believe that exhaustivity will have a profound impact on the industry, well beyond the two use cases mentioned in this article.