THE “BLACK BOX” ISSUE: ALGORITHMIC OPACITY, AUTOMATED DECISION-MAKING AND CONSTITUTIONAL RESPONSIBILITY IN INDIA
CHAPTER 1
STATING THE “BLACK BOX” PROBLEM: AADHAAR AUTHENTICATION AND CONSTITUTIONAL RESPONSIBILITY
1.1 Introduction
Technology plays an increasingly prominent role in governance. Governments are depending on digital technology to identify people, verify information, distribute benefits, prevent fraud and make administration more efficient. And while the use of digital technology is often characterised as a move away from slow and unpredictable human administration towards objective and standardised technological decision-making, widespread usage of automated decision-making tools poses a constitutional question which cannot be determined simply by whether the technology itself is efficient and accurate. When a technological system produces a decision which impacts a person’s legal rights, access to public benefits or exercise of a protected right, it must be asked whether that person can understand the justification for the decision and challenge it effectively.
The problem is often summed up in the concept of the “black box”. In broad terms, a black box exists where the relationship between information input into a system and the response (or output) generated by that system is insufficiently transparent to the individual affected by that output. The affected individual can be aware that a certain application has been rejected, authentication has been denied, or an individual has been assigned a certain classification or rating, but may be unaware of the rationale behind that result. Algorithmic opacity does not mean that technology is necessarily wholly secret
. Governments may publish the general purpose of an algorithm, outline its legal framework generally, or even provide technical details without allowing the individual to ascertain why a different decision was made in their particular circumstances.
This issue is more pertinent where the State is involved. While an algorithm may be employed to suggest a product to a private company or to screen through information, the position in relation to the State in reliance on this technology is different as the State employs it during the course of the exercise of public authority. The State’s decisions are governed by the constitutional constraints imposed in terms of equality, lack of arbitrariness, fairness, privacy, and in the case of judges, the question of the review of their decisions. Simply because a computer is employed does not mean that a failure to adhere to these obligations cannot occur; if anything, automating these duties may increase its importance, as the individual affected may be even less able to identify who in the system is responsible for the negative outcome.
Indian constitutional law does not explicitly deploy the language of algorithms or automated decision-making, (or indeed of algorithmic accountability) but the Constitution is well equipped to do so through its doctrinal framework. State action must meet the challenge of being both non-arbitrary and consistent with equality under Article 14. Fair, just and reasonable procedure must be upheld under Article 21, which safeguards dignity, privacy and autonomy. The recent Constitution Bench decision on Justice K.S. Puttaswamy (Retd.) v. Union of India made the recognition of privacy as a fundamental right as well as the constitutional importance of informational privacy the cornerstone of a whole new set of protections for individuals. Natural justice, reasoned decision-making, and the reviewability of State action, all are built-in checks on the reasonableness of the State’s action, and that include requirements of a fair opportunity to be heard.
This chapter analyzes the problem via Aadhaar authentication. Aadhaar does not involve the use of artificial intelligence in the traditional sense nor is the purpose of this research to categorize Aadhaar as an artificial intelligence system. It is used here as a definitive Indian example of an automated public system whereby information provided by a user or individual is first submitted to a technological infrastructure and subsequently generates a legally relevant output. Under the Aadhaar system, the biometric, demographic or other form of information submitted by the individual is verified with the UIDAI system and an authentication response is generated.
In terms of the technology itself this might seem a relatively tight identity-verification process. But at the level of constitutional law, it becomes a matter of significance depending on what flows from that outcome. If the individual is then required to repeat the process and authentication fails, the consequence is trivial. But if the same failure is employed to refuse a pension, a subsidy, a food entitlement or some other form of public service, then there is a legal significance to that technological output.
Thus, on this reading, the real constitutional issue at stake is not whether technology should be used by the State, but whether such use shifts the parameters of constitutional criticism for State action. My contention is that, irrespective of the answers to the other questions raised, it should not. Whether it is applied as a result of automation or not, the State cannot escape constitutional accountability.
1.2 Understanding Algorithmic Opacity
Although the term ‘black box’ is applied indiscriminately, it is helpful to delineate some different kinds of opacity. An algorithm may be opaque because it has been secreted away by the state or corporation, because the technical processes are too complicated to be meaningful for a layperson, or because the user simply does not have the access to the information necessary to know whether the effect is justified. The two types of opacity above may occur simultaneously, and the crucial consideration here is that transparency should not be conflated with open source software.
This distinction between transparency and explanation is especially significant for public administration. The state might publish rules that say, for example, “When you need to check someone’s identity, we use biometric recognition”. The person using the system may never understand-may never have reason to understand-the details of why a particular person was not successfully recognised. Even if the person learns that there was a “no match” response, they may not understand the technical cause, which could be incomplete enrolment data, low quality data, network congestion, or some other cause. This is transparency at the level of general description but opacity at the level of effect on constitutionally protected rights.
This also creates an institutional problem. In traditional administrative decision-making, the person who is taking the decision is normally a distinct entity, who can and should be asked to give reasons for the decision in question. Automated decision-making blurs this relationship. For a technological result, there may be statutory authority, a government department, database, authentication service provider, and other technical actors involved. The individual may perceive all of this as one State decision, but the institutional responsibility for the outcome is divided and if something goes wrong the responsibility gap arises: everyone can explain part of the process but no institution is responsible.
The black box issue is therefore not just about technology being hard to comprehend. It is also about the interplay between information, decision-making and accountability. A constitutional system should stay able to explain why a decision was taken, who took it and how it can be challenged.
1.3 Aadhaar Authentication as a Constitutional Case Study
The example of Aadhaar is especially relevant as it involves use of technology-enabled authentication but the impact is felt in public administration. The Aadhaar Act, 2016 sets the legal framework for the issue of Aadhaar numbers and for the authentication process. The Aadhaar (Authentication and Offline Verification) Regulations set out the operational procedures for authentication, including sections pertaining to the consent of Aadhaar holders, use of information, security and authentication records.
The binary response is the output of the authentication process. The binary nature of the response is not indicative of the simplicity of the authentication process.
