By Dr Stephen Castell CITP CPhys FIMA MEWI MIoD
Challenging the reliability of AI and other complex multi-connected intelligent computer systems will become of fundamental importance as businesses and society move rapidly towards a software-dominated, algorithm-governed future. Increasingly ubiquitous ‘algo dependency’ is likely to result in a wide variety of disputes, some of which will reach court, in which computer evidence will critically feature.
Such conflicts have indeed already arrived: in a recent article [1] I set out the substance of my sworn expert witness testimony given in a recent USA FINRA Arbitration. This was for a dispute over use of Automated Decision Systems (ADS) by a major US fund management corporation to liquidate the investment position of a client, with heavy losses. As the use of such ADS spreads widely, and courtroom battles follow, the costs of employing legal and computer experts in their resolution – in particular with regard to discovery of digital evidence and its presentation, explanation and validation in court – will be small compared with the profound financial impact of presuming that the automated functionality and operations of AI, and evidence derived from ADS, can be trusted as being reliable.
But why should there be any such presumption of the reliability of computer evidence? It is interesting that it seems the question does not even arise in the USA, where, for example, Arizona is a leader in handling digital evidence [2]; and, further afield in North America, it was noted in 2003 [3], “The ability to convince the court that digital evidence is worthy of reception into the criminal process is dependent on the qualifications and competence of the tendered expert, the skill and knowledge of the prosecutor in leading such evidence and the quality of the digital evidence itself. Digital evidence has been successfully led in many jurisdictions in the United States and Canada over the past decade”. There is here no hint of a ‘presumption’, but, correctly, emphasis is placed on the diligence, skill, competence and experience of legal and technical professionals with due regard to the special nature of computer evidence, in properly revealing and demonstrating its reliability, quality and relevance (or of course challenging the computer evidence adduced, as the case may be).
In the UK, however, there is a rather different jurisprudential situation. Over many years, I have published extensively on the questionable presumption of the reliability of computer evidence, beginning with work commissioned by H M Treasury on the admissibility of computer evidence in court and the legal admissibility, reliability and security of IT systems. Published in 1990 by permission of HMG [4], that work was definitive in the critical field of the legal standing and trustworthiness of computer software and systems, and of electronic evidence derived from such systems. My recommendations included that, in court trials where computer evidence is to be adduced, there should be appointed an Independent Technical Assessor to assist the Judge.
Almost certainly on grounds of perceived cost-saving to the National Exchequer, my 1990 recommendations were not fully implemented by the UK government. To the contrary, and to the astonishment of concerned expert ICT and legal professionals, an incorrect ‘presumption of the reliability of computer evidence’ later passed into UK law. This, still current, legal rule was introduced in 1999 after a Law Commission recommendation that courts presume a computer system has operated correctly unless there is explicit evidence to the contrary. This replaced Section 69 of the 1984 Police and Criminal Evidence Act (PACE), which stated that computer-based evidence should be subject to proof that the computer system was operating properly. Reliance on this 1999 ‘presumption’ legal rule subsequently became common in pleadings, and accepted by presiding judges, albeit principally in criminal prosecutions (and rarely in civil disputes, where, in my long experience as an expert witness in UK, and USA, courts, all relevant evidence is disclosable, and routinely probed, ferociously).
In particular, this ‘presumption’ was a major factor in the 20-year-long faulty UK Post Office Horizon system prosecutions scandal, the most widespread miscarriage of justice in recent British legal history [5]. The Horizon debacle begs the question: why was it thought that ignoring those 1990 computer evidence recommendations would save costs in the justice system? It is clear that the costs simply of the several hundred PO Horizon prosecutions, and the financial – and life – consequences to the hundreds thereby falsely accused and convicted, plus the further ongoing costs of the current Sir Wyn Williams Public Inquiry into PO Horizon, and the compensation payments that could result, vastly dwarf the relatively modest costs of implementing my 1990 recommendations for computer experts to be intrinsically woven, as court-appointed Independent Technical Assessors, into UK justice processes.
Furthermore, the PO Horizon fiasco was due as much to the paucity of Legal Aid funding for defendants. Proper funding would have allowed ICT expert witnesses to demand discovery of, and competently investigate, the relevant software and systems and provide testimony to the court on a defendant’s behalf. It is true that the ‘presumption of reliability’ is in principle rebuttable, but that rarely happens effectively in practice, since the defendant has meagre resources compared with the state, the prosecution. I myself was approached to be a defendant’s expert for one of the very first Post Office Horizon system cases in 1999, but the accused Sub-Postmistress’s solicitors could not obtain Legal Aid funding to engage me.
Once again, saving costs in the justice system has simply led to massively increased costs and consequential financial damage later – even excluding potential future reparations for the shocking failure to deliver justice to the ‘Horizon hundreds’ falsely accused. This inadequate funding of and in the UK justice system has also reached a crisis point amongst barristers themselves, with Members of the Criminal Bar Association (CBA) refusing to take on new cases from Monday June 27, 2022, staging a first whole week of strike action on Monday July 18, 2022, and saying such action will persist and possibly escalate, if a deal over proper levels of legal aid funding were not to be agreed with the government [6].
I have recommended that the current Horizon Inquiry address this Legal Aid funding lacuna; and that the Inquiry should also be guided by experienced independent professional forensic ICT experts. I contend that the Inquiry must take on board over thirty years of scholarly and practitioner literature on electronic documents, plus ‘litigation trenches’ protagonist experience, concerning the legal and technical falsity of the ‘presumption of the reliability of computer evidence’. Removal of this faulty UK justicial presumption is well overdue. The bottom-line is that the presumption is – always was – simply wrong, and if only on costs grounds alone, it should not be kept in place.
Beyond the formal forensic procedural setting, it is timely that the wider ICT profession is itself grappling with asking the right questions of computer evidence, of AI, and of the ever-escalating deployment of interconnected software and systems. The IT Leaders Forum of the British Computer Society (BCS, the UK’s Chartered IT Institute) has set up a Software Risk and Resilience Working Group [7]. This is intended to provide a framework for action should a major crisis be caused by a software disaster – of a scale vastly more massive in its consequences than PO Horizon – aiming to establish understanding of potential risks and resilience factors in infrastructure software in the UK.
It is clear that society’s growing ubiquitous dependencies on software, and on national-scale ADS, pose significant new issues as to the robustness, rigour and legality of the design and implementation of such systems. In particular, the definition of ADS functional requirements, and the ‘baked-in’ bias in their algorithms, is of fundamental concern to citizens. The computer evidence pertaining thereto should clearly be open to independent democratic audit – and certainly never ‘presumed reliable’. For example, I suggested in a recent letter published in the Financial Times that the national economic anti-inflation algorithm to which HM Treasury is currently working should be published, in detail, for citizens and experts to examine and assess, and upon which to give their views, as a matter of best practice and sound algorithmic governance at the national level [8].
© 2022 Stephen Castell
About the Author
Dr Stephen Castell, an award-winning ICT systems and software consultant professional, and FinTech visionary, has for decades been active as an international expert witness in major complex computer software and systems disputes and litigation, including the largest and longest such actions to have reached the English High Court, and Sydney Supreme Court.
References
[1] Castell, S., 2022. “New Financial Risks Arising from Digital Finance: Disputes Over Automated Decision Systems and Algorithmic Assessments by ICT Forensic Expert Witnesses”, Acta Scientific Computer Sciences, Volume 4 Issue 7 – 2022 (Published July 01, 2022): 24-36.
https://www.actascientific.com/ASCS-4-7.php
https://www.actascientific.com/ASCS/pdf/ASCS-04-0291.pdf
https://scholar.google.com/citations?view_op=list_works&hl=en&hl=en&user=TJHeIBsAAAAJ
[2] (i) Blye, A., 2021. “Arizona to become first state to move all courts to digital evidence management with Thomson Reuters”, Phoenix Business Journal, January 22, 2021.
(ii) Digital Evidence: The Arizona Supreme Court has launched a digital evidence portal that provides a standard solution for the submission, management, and storage of case exhibits. The digital evidence portal provides convenient 24/7 online access to upload, review, and manage exhibits in a case which a party has been invited. …
https://www.azcourts.gov/digitalevidence/
[3] Hak, J. W., 2003. “The Admissibility of Digital Evidence in Criminal Prosecutions”, Jonathan W. Hak, Crown Prosecutor, Department of Justice, Alberta, Canada, January 2003.
https://www.crime-scene-investigator.net/admissibilitydigitaleveidencecriminalprosecutions.html
[4] Castell, S., 1990. The APPEAL Report, May, Eclipse Publications, ISBN 1-870771-03-6.
[5] The UK Post Office Horizon system scandal:
(i) Meddings, S., 2022. “Should those who sent the sub-postmasters to prison now face court themselves? Sub-postmasters were hounded for years, even after doubts were raised over the IT evidence that convicted them. Despite this, their prosecutors remain unrepentant”, The Times, Saturday August 06 2022.
(ii) Flinders, K, 2022. “Government has no plans to review controversial court rules on computer evidence. Government accused of ‘passing the buck’ and ‘not knowing what it is talking about’ …”, Computer Weekly, 23 May 2022.
(iii) Castell, S., 2021. “A trial relying on computer evidence should start with a trial of the computer evidence. Learning from the Post Office Horizon scandal …”, Computer Weekly, 22 December 2021.
(iv) Mawardi, A., 2021. “Mishcon and Eversheds among firms called up for Post Office inquiry”, The Lawyer, 12 October 2021.https://www.thelawyer.com/mishcon-and-eversheds-among-firms-called-up-for-post-office-inquiry/
(v) Hyde, J., 2021. “News focus: Post Office Horizon scandal – Where weren’t the lawyers? There are calls for a full, judge-led public inquiry into the Post Office prosecution of former subpostmasters, with many wondering what role lawyers played in this miscarriage of justice…”, The Law Society Gazette, 3 May 2021.
[6] Siddique, H., 2022. “Barristers in England and Wales stage first five-day strike over legal aid funding”, The Guardian, Monday 18 July 2022.
[7] Ringland, G., Steinmueller, E., and Castell, S., 2022, “BCS IT Leaders Forum Software Risk and Resilience Working Group”. “Terms of Reference for the Working Group …
- In the short term, create a network of people and organisations with an understanding of software risks and their potential impact.….
- In the longer term the aim is … to provide a framework for action to reduce the impact of software failures on the UK economy …”. Further details available from:
Gill Ringland ([email protected]), Professor Ed Steinmueller ([email protected]), Dr Stephen Castell ([email protected]) or Dr David Miller, Chair ([email protected]), BCS IT Leaders Forum
https://www.bcs.org/membership-and-registrations/member-communities/bcs-it-leaders-forum/
[8] Castell, S., 2022. Letter: “Thatcher’s acronyms seem ready-made for new leader”, Financial Times, Friday, 22 July, 2022, page 22.
https://www.ft.com/content/15b405d2-9fd7-482f-bbdb-41e55979d34d
Background Reading
Castell, S., 2021. “Direct Government by Algorithm Towards Establishing and Maintaining Trust when Artificial Intelligence Makes the Law: a New Algorithmic Trust Compact with the People”, Acta Scientific Computer Sciences 3.12 (2021): 04-21, November 10, 2021.
https://www.actascientific.com/ASCS/ASCS-03-0194.php
Castell, S., 2021. “Block Chains and Stablecoins”, ExpertWitness.co.uk, January 7, 2021.
https://www.expertwitness.co.uk/articles/news/mr-s-castell-block-chains-and-stablecoins
Mason, S.. and Seng, D., editors, 2021. “Electronic Evidence and Electronic Signatures” (5th edition), Institute of Advanced Legal Studies for the SAS Humanities Digital Library, School of Advanced Study, University of London, 2021 https://humanities-digital-library.org/index.php/hdl/catalog/book/electronic-evidence-and-electronic-signatures
Castell, S., 2021. “Blockchain and Cryptocurrency Tracing Disputes: Digital Forensics Evidential Standards”, Experts.com, articles https://www.experts.com/articles/blockchain-cryptocurrency-tracing-disputes-digital-forensics-evidence-by-dr-stephen-castell
Rishwain, N, 2020. “Is New Hampshire the Next Iowa Voting Disaster? Information Technology Expert Analysis”, Experts.com, February 7, 2020.
Castell, S., 2006. “Forensic Systems Analysis: A Methodology for Assessment and Avoidance of IT Disasters and Disputes”, issued as a Cutter Consortium Executive Report, Enterprise Risk Management & Governance Advisory Service series, Vol. 3, No. 2, 8 March 2006.
Castell, S., 2003. “Role of the IT expert witness: The role of the IT expert witness in software and systems development/implementation contract disputes and litigation”, Computer Law & Security Review, 19(3):228–231, May 2003.
Castell, S., 1996. “Seeking after the truth in computer evidence: any proof of ATM fraud?”, The Computer Bulletin, Volume 38, Issue 6, December 1996, Pages 17–19.
https://academic.oup.com/itnow/article-abstract/38/6/17/374876
Castell, S., 1994. “A computer of the simplest kind”, Computer Law and Security Report, 10, May-June 1994 (and see further references under its ‘FOOTNOTES’).
Castell, S., 1993. “Computers trusted, and found wanting”, Computer Law and Security Report, 9, July-August 1993, pp. 155-156.
Castell, S., 1991. “Computers out of control”, Financial Times, May 29, 1991, Wednesday, SECTION I; Letters, Pg. 19.
Dr Stephen Castell CITP CPhys FIMA MEWI MIoD
Chairman, CASTELL Consulting
PO Box 334, Witham, Essex CM8 3LP, UK
Tel: +44 1621 891 776 Mob: +44 7831 349 162
Email: [email protected]
http://www.e-expertwitness.co.uk
https://archivesit.org.uk/interviews/stephen-castell/
