Indian Statistical System - Report

Indian Statistical System - Report

Revamping India's statistical system is crucial for its economic future. The system, once a source of global repute, requires changes now to keep up with the emerging scenario . To navigate the country's economic and social challenges, we need to urgently invest in a modern, credible, and independent statistical framework.

The current system suffers from a set of challenges. Persistent delays and leave policymakers and businesses operating in the dark. The lack of disaggregated data at the state and district levels means that one-size-fits-all policies are made for a diverse country with vast regional disparities.

The Indian statistical system faces a wide array of challenges that hinder its ability to provide accurate, timely, and reliable data for effective governance and informed public discourse. These issues range from fundamental problems in data collection and methodology to systemic and institutional weaknesses.

Data Inconsistency and Credibility Issues

The credibility of India's statistical system has been significantly eroded by frequent revisions of key economic indicators, such as GDP growth rates, and noticeable discrepancies between data from different government sources. This lack of a single, coherent narrative undermines public and investor trust in the official data.

Delays in Data Collection and Dissemination

A major challenge is the considerable time lag between data collection and release. Critical surveys, like those on employment and household consumption, are often published years after the data is collected, making the information stale and less effective for addressing current policy needs.

Outdated Methodologies

India's statistical system still relies heavily on traditional, manual survey methods, which were designed for a different era. These methodologies struggle to accurately capture the complexities of a modern, digitized economy, including the rise of the service sector, e-commerce, and the gig economy.

Lack of Real-time Data

For a rapidly evolving economy, the absence of high-frequency, real-time data is a significant handicap. Policymakers need up-to-the-minute information on key economic and social indicators to respond quickly to dynamic issues like inflation spikes or unemployment trends.

Weak Institutional Autonomy

The National Statistical Commission (NSC) and the National Statistical Office (NSO) lack the full autonomy necessary to operate free from external pressures.

Inadequate Coordination

Poor coordination among central ministries, state statistical bureaus, and various data-collecting agencies results in duplicated efforts and inconsistent datasets. This fragmentation leads to a disjointed statistical landscape where different bodies produce conflicting figures on the same subject.

Insufficient Funding and Human Resources

Statistical departments are often underfunded and face a severe shortage of skilled personnel. This lack of investment hampers their ability to conduct large-scale, comprehensive surveys and adopt modern technologies, further perpetuating the system's reliance on outdated methods.

Poor Data Granularity

Many surveys provide only broad national aggregates, which are insufficient for targeted policy interventions. There is a critical need for more granular, disaggregated data that can shed light on regional disparities, gender inequalities, and caste-based differences at the district or sub-district level.

Challenges in Measuring the Informal Sector

India's economy has a massive informal sector, which is very difficult to measure. The lack of accurate data on informal enterprises and employment leads to flawed estimates of GDP, labour force participation, and overall economic activity, presenting a distorted picture of the economy.

Lack of a Legal Framework

The absence of a robust legal framework for official statistics is a major systemic weakness. This leaves the system without a clear, legally binding mandate for data collection and dissemination, and without clear policy guidelines, hindering accountability and transparency.

Poor Data Quality

The quality of administrative data, which is generated by various government departments as part of their routine functions, is often poor. This data is not standardized and contains significant errors and inconsistencies, making it difficult to integrate into the official statistical system for use.

Gaps in Data Coverage

Significant data gaps exist in critical sectors like employment, health, and social indicators. For example, India has not released official poverty estimates since the 2011-12 consumption survey, leaving a decade-long vacuum in a crucial area of public policy.

Technological Lag

The Indian statistical system has been slow to adopt modern technologies like big data analytics, AI, machine learning, and cloud computing. This technological lag prevents it from processing large datasets efficiently and from moving beyond traditional survey methods.

Sub-optimal Use of Administrative Data

India generates vast amounts of administrative data from schemes like the GST, Aadhaar, and various government portals. However, this data is not effectively integrated into the official statistical system, representing a missed opportunity to supplement and validate traditional survey data.

Lack of Data Transparency

The methodologies used for data collection and the raw data from surveys are not always made publicly available. This lack of transparency hinders independent verification, analysis, and research, making it difficult for academics and experts to scrutinize the official figures.

Inadequate Training and Capacity Building

The skills of government statisticians often do not keep pace with global advancements in data science and technology. There is a significant need for continuous training and capacity-building programs to equip them with the tools and knowledge needed for a modern statistical system.

Shortage of a Statistical Cadre

India faces a shortage of a dedicated, professionally trained cadre of statisticians. Many key positions are filled by general bureaucrats who may lack the specialized skills and expertise required for rigorous statistical work, impacting the quality of data collection and analysis.

Poor Data Dissemination

Official data is often not published in user-friendly formats, such as APIs or interactive dashboards. It is frequently released in static, difficult-to-parse formats like PDFs, making it challenging for researchers, policymakers, and the public to access, analyse, and utilize the information.

Over-reliance on Estimates

The statistical system often relies on extrapolating data from small sample sizes to produce national-level estimates. This can introduce significant inaccuracies and sampling errors, particularly when the sample size is not representative or is not properly "blown up" to reflect the diversity and scale of the country.

Action Taken by the Government 

Government has initiated several measures to address these challenges, though progress has been mixed.

Institutional Reforms

The government has undertaken significant institutional reforms to streamline the statistical system. A major step was the restructuring of the Ministry of Statistics and Programme Implementation (MoSPI) and the creation of the National Statistical Office (NSO) in 2019. The NSO was formed by merging the two primary bodies: the National Sample Survey Office (NSSO), which was responsible for large-scale sample surveys, and the Central Statistical Office (CSO), which handled national accounts and industrial statistics. This consolidation was intended to improve coordination and avoid data discrepancies by bringing key functions under a single umbrella.

National Statistical Commission (NSC)

The government established the National Statistical Commission (NSC) as an advisory body to the government. The primary goal of the NSC is to bring a higher degree of professionalism and scientific standards to the collection, analysis, and dissemination of official statistics. By providing a platform for independent expert advice, the NSC aims to enhance the credibility and autonomy of the statistical system. However, its effectiveness has been debated, with some critics pointing to its limited executive powers and the fact that its recommendations are not always binding.

Support for Statistical Strengthening (SSS) Scheme

MoSPI launched the Support for Statistical Strengthening (SSS) Scheme to address the weaknesses in state-level statistical systems. Under this scheme, the central government provides financial and technical assistance to state statistical bureaus to improve their infrastructure, data collection methods, and human resources. The objective is to ensure that state-level data is collected more uniformly and to a higher standard, which is crucial for producing accurate national aggregates and for enabling granular, district-level analysis.

Adoption of Technology

To improve the timeliness and accuracy of data collection, the government has pushed for the adoption of technology. This includes the use of tablets for Computer-Assisted Personal Interviewing (CAPI) in national surveys. Instead of using traditional paper questionnaires, surveyors now use digital devices to record responses. This move reduces errors, accelerates data submission, and allows for near real-time monitoring of survey progress.

Draft National Policy on Official Statistics (NPOS)

Recognizing the need for a comprehensive legal framework, MoSPI has proposed the Draft National Policy on Official Statistics (NPOS). This policy aims to provide a clear legal and institutional framework for the statistical system. The NPOS is designed to modernize the system, ensure data quality, and clarify the roles and responsibilities of various data-collecting agencies. It also seeks to establish a more transparent and credible system that can meet the demands of a modern economy.

Utilizing Administrative Data

The government has made efforts to utilize administrative data from various government programs and databases. A key example is the ongoing initiative to integrate data from the Goods and Services Tax Network (GSTN) into national accounts and economic statistics. By using GSTN data, the government can get a more accurate and timely picture of formal sector economic activity, reducing the reliance on traditional, time-consuming surveys. This is part of a broader push to leverage the vast amounts of digital data being generated by government schemes.

Release Calendars

To improve timeliness and predictability, the government has made it a practice to publish advance release calendars for key statistical reports. This allows policymakers, researchers, and the public to know when important data, such as GDP figures or inflation numbers, will be released. This move enhances transparency and helps to mitigate concerns about data being withheld or delayed for political reasons.

Strategies for improvement

A comprehensive action plan to revamp the Indian statistical system can address institutional, technological, and human resource challenges to restore credibility and provide reliable data for a modern economy.

Strengthen Institutional Autonomy

A foundational step is to make the National Statistical Commission (NSC) a permanent, statutory, and fully autonomous body. It could have the legal authority to audit and certify data quality, ensuring all official statistics meet consistent standards. By operating independently , the NSC can ensure  the objectivity of data.

Modernize Data Collection

The system can transition from outdated, paper-based surveys to a fully digital, real-time data collection system. Using technologies like Computer-Assisted Personal Interviewing (CAPI) and mobile apps allows field surveyors to instantly transmit data, drastically reducing time lags and improving accuracy. This shift is crucial for providing timely insights for dynamic policymaking.

Establish a National Data Warehouse

Creating a centralized, secure data warehouse for all official statistics is vital. This repository would provide a single, reliable source of information, eliminating inconsistencies and making data easily accessible to researchers, policymakers, and the public, while ensuring security and privacy.

Integrate Big Data and AI

The government can effectively utilize the vast amounts of administrative data generated by sources like the Goods and Services Tax Network (GSTN) and the Unified Payments Interface (UPI). By using advanced analytics and Artificial Intelligence (AI), it can generate more granular and real-time insights than traditional surveys alone.

Conduct a Decennial Census

The decennial population census is the foundational baseline for all official statistics. Prioritizing and conducting the 2021 census without further delay is critical to update demographic and socio-economic data, which are vital for policymaking, resource allocation, and providing a reliable framework for all other surveys.

Create a Statistical Reforms Commission

Constituting a high-level, time-bound Statistical Reforms Commission is necessary to diagnose deep-rooted systemic issues. This body could have a clear mandate to recommend a new, robust legal framework to govern the entire statistical system and restore its integrity.

Overhaul Human Resources

The system needs a major overhaul of its human resources. This involves recruiting and training a dedicated, professional Indian Statistical Service (ISS) cadre and providing continuous skill development in modern data science, analytics, and data governance. This ensures the workforce can handle new technologies and complex datasets.

Increase Funding

Significantly increasing budgetary allocations for statistical agencies is essential. This funding is crucial for investing in modern technology, building necessary infrastructure, and attracting and retaining skilled personnel, all of which are critical for the system's long-term sustainability.

Improve Data Granularity

Future surveys can be designed to collect disaggregated data at the state, district, and sub-district levels. This level of detail is essential for creating targeted policy interventions that address specific regional disparities or social issues, rather than relying on broad national averages.

Bridge Data Gaps

New surveys can be launched or existing ones revamped to collect timely and reliable data on key indicators where significant gaps exist, especially for employment and household consumption. This would provide policymakers with up-to-date information on the economy and social welfare.

Enhance Data Transparency

To rebuild trust, it's vital to mandate the public release of all anonymized raw data and detailed survey methodologies. This transparency would enable independent scrutiny and academic research, allowing for data validation and new analysis.

Standardize Data Protocols

Implementing uniform standards and protocols across all central ministries and state governments for data collection and storage is crucial. This would ensure consistency, interoperability, and the ability to compare data from different sources without discrepancies.

Strengthen State Statistical Systems

The central government can provide additional funds and technical support to state-level statistical bureaus. Strengthening these systems is crucial because much of the ground-level data is collected at the state and local levels, impacting the quality of national aggregates.

Develop a National Statistical Policy

Enacting a new, legally binding National Policy on Official Statistics is a foundational step. This policy would provide a comprehensive legal framework to govern all aspects of the statistical system, from data collection and quality standards to dissemination protocols and accountability.

Engage with Stakeholders

Creating a formal mechanism for continuous feedback from data users, including economists, researchers, the private sector, and civil society, is essential. This ensures the system produces data that is relevant and useful to a wide range of users.

Establish a Data Quality Assurance System

Implementing a rigorous, third-party audit system is necessary to verify the quality and reliability of all official statistics. An independent body could perform regular checks to ensure data accuracy and adherence to best practices, enhancing the system’s credibility.

Measure the Informal Sector

Developing innovative and robust methodologies is critical to accurately measure economic activity in the informal and unorganized sectors. This could involve using a combination of survey data, administrative records, and new technologies to capture this hard-to-measure part of the economy.

Modernize National Accounts

The methodology for calculating key macroeconomic indicators like GDP needs to be updated to align with international standards and reflect the modernizing Indian economy, including the digital and gig economies, for a more accurate representation of economic activity.

Launch a Public Awareness Campaign

A public awareness campaign could be launched to educate people on the importance of statistics and the methodology behind them. This would not only help rebuild public trust but also encourage greater participation in surveys, leading to better data quality.

Foster International Collaboration

Collaborating with international statistical organizations and academic institutions can help India adopt global best practices, learn from other countries' experiences, and modernize its methodologies. This will ensure that India's statistical system is on par with global standards.

Comparison of the statistical systems of India, the US, the UK, and Japan

 

Structure & Organization

 

India: Has a decentralized statistical system. The Ministry of Statistics and Programme Implementation (MoSPI) is the nodal agency, and it contains the National Statistical Office (NSO), which consists of the Central Statistical Office (CSO) and the National Sample Survey Office (NSSO). State-level Directorates of Economics & Statistics (DES) exist for coordination.

US: Highly decentralized. Over 100 agencies are involved, with 13 Principal Statistical Agencies (PSAs) responsible for the bulk of official data. Coordination is managed by the Office of Management and Budget (OMB) and the Chief Statistician of the United States.

UK: A mix of centralized and decentralized. The UK Statistics Authority (UKSA) oversees the system, with the Office for National Statistics (ONS) as the main producer of statistics. Other government departments also produce statistics.

Japan: Decentralized. While the Statistics Bureau of Japan (SBJ) under the Ministry of Internal Affairs and Communications (MIC) conducts fundamental surveys, various other ministries have their own statistical departments.

Legal Framework

 

India: Governed by the Collection of Statistics Act, 2008, which allows the government to collect statistics on a wide range of subjects. Other specific acts, like the Census Act, 1948, also exist.

US: Has multiple laws. The Confidential Information Protection and Statistical Efficiency Act (CIPSEA) of 2018 is a key piece of legislation that ensures confidentiality and allows for data sharing among statistical agencies.

UK: The Statistics and Registration Service Act 2007 established the UKSA and its legal authority to produce and regulate statistics.

Japan: The Statistics Act (2007) provides the legal basis for the production of official statistics and outlines the roles of various statistical bodies.

Key Nodal Agency

 

India: Ministry of Statistics and Programme Implementation (MoSPI).

US: Office of Management and Budget (OMB), specifically the Office of Information and Regulatory Affairs (OIRA).

UK: UK Statistics Authority (UKSA).

Japan: Statistics Bureau of Japan (SBJ) under the Ministry of Internal Affairs and Communications (MIC).

Primary Data Producers

India: The National Statistical Office (NSO), which includes the Central Statistical Office (CSO) and the National Sample Survey Office (NSSO).

US: The 13 Principal Statistical Agencies, including the Bureau of Economic Analysis (BEA), Bureau of Labor Statistics (BLS), and the Census Bureau.

UK: The Office for National Statistics (ONS) is the largest producer, with other government departments also producing statistics.

Japan: The Statistics Bureau of Japan (SBJ) and statistical departments within other ministries.

 

Role of a Chief Statistician

 

India: The Secretary of MoSPI holds a similar role but lacks the statutory independence of counterparts in other countries.

US: The Chief Statistician of the United States in the OMB provides oversight and coordination.

UK: The National Statistician is the Head of the Government Statistical Service (GSS) and is the Chief Executive of the ONS.

Japan: The Director-General of the Statistics Bureau serves as the head of Japan's statistical system.

 

Coordination

 

India: Coordination happens through various committees and the NSO, but it can be challenging due to the decentralized structure and limited statutory power of the National Statistical Commission (NSC).

US: Coordinated by the OMB through interagency councils like the Interagency Council on Statistical Policy (ICSP).

UK: Coordination is facilitated by the UKSA and the Government Statistical Service (GSS), a network of all official statisticians.

Japan: Coordination is managed by the Statistics Commission, which provides advice and oversight to the MIC.

 

Statistical Independence

 

India: The statistical system has faced concerns about its autonomy and potential interference in data releases. The National Statistical Commission (NSC) is an advisory body but lacks statutory authority to enforce its recommendations.

US: The decentralized structure and the Evidence-Based Policymaking Act of 2018 provide a high degree of independence. Agencies are legally mandated to produce and disseminate statistics without political interference.

UK: The UKSA is an independent body that ensures the production of impartial, high-quality statistics, promoting public trust. The Code of Practice for Statistics is a key tool for ensuring integrity.

Japan: The Statistics Act and a well-established civil service tradition provide a strong framework for statistical independence.

 

Quality Assurance & Standards

 

India: The National Statistical Commission (NSC) sets standards, but their enforcement principles vary.

US: Standards and guidelines are set by the OMB and enforced across federal statistical agencies.

UK: The Office for Statistics Regulation (OSR), the regulatory arm of the UKSA, assesses official statistics against the Code of Practice for Statistics.

Japan: The Statistics Commission oversees the quality and methodology of statistical surveys.

 

Major Statistical Outputs

 

India: Key outputs include the National Accounts Statistics (NAS), Index of Industrial Production (IIP), Consumer Price Index (CPI), and results from the Annual Survey of Industries (ASI) and various socio-economic surveys.

US: Major outputs include the Gross Domestic Product (GDP), Consumer Price Index (CPI), Unemployment Rate, and the Current Population Survey (CPS).

UK: Key outputs include the Consumer Price Index (CPI), GDP, Labour Market Statistics, and the UK Census.

Japan: Major outputs include the Consumer Price Index (CPI), Labour Force Survey, and the Population Census.

Data Accessibility

 

India: Data is available on the MoSPI website and various ministry portals, but it can sometimes be fragmented and lack comprehensive metadata.

US: Data is highly accessible through various agency websites and a central portal, data.gov. There is a strong emphasis on providing microdata for research.

UK: Data is readily available on the ONS website, with many datasets open to the public. There are also secure data research environments for more sensitive data.

Japan: Data is provided through the Portal Site of Official Statistics of Japan (e-Stat), which offers a consolidated platform.

 

Data Confidentiality

 

India: The Collection of Statistics Act provides for data confidentiality, with strong penalties for misuse.

US: The CIPSEA and other laws provide robust protections for confidential information collected by statistical agencies.

UK: Statistical agencies adhere to strict data protection and privacy laws, including the General Data Protection Regulation (GDPR).

Japan: The Statistics Act ensures the confidentiality of information collected for statistical purposes.

 

Use of Administrative Data

 

India: Efforts are being made to increase the use of administrative data, but the integration and quality remain a challenge.

US: Statistical agencies are increasingly using administrative data to reduce survey costs and respondent burden.

UK: The use of administrative data is a key part of the statistical system's strategy to produce more timely and efficient statistics.

Japan: Administrative data is used, but a heavy reliance on traditional surveys remains.

 

Census & Surveys

 

India: The Population Census is conducted every 10 years by the Office of the Registrar General and Census Commissioner. The NSSO conducts large-scale household and enterprise surveys.

US: The Census Bureau conducts the decennial census and other large-scale surveys like the American Community Survey (ACS).

UK: The ONS conducts the UK Census every 10 years.

Japan: The SBJ is responsible for the Population Census, conducted every five years, and other major surveys.

 

Human Resources

 

India: Has a dedicated Indian Statistical Service (ISS) cadre for statisticians.

US: Relies on a combination of federal employees, contractors, and academic professionals.

UK: Has a Government Statistical Service (GSS), a professional body for all statisticians working in government.

Japan: The statistical system employs a dedicated workforce of statisticians and is supported by a robust training institute.

International Collaboration

 

India: Collaborates with international organizations and participates in global initiatives, but this could be strengthened.

US: A major player in international statistical collaboration, often setting global standards and providing technical assistance.

UK: Actively involved in international bodies and initiatives, sharing best practices and data.

Japan: A significant contributor to international statistical cooperation, especially in Asia.

The Path Forward

The government has initiated some reforms, but further reforms are being implemented. The nation can prioritize a legally binding National Policy on Official Statistics that provides a clear roadmap for reform. We can also launch a public awareness campaign to educate citizens on the importance of statistics and encourage their participation, helping to rebuild the bridge of trust between the system and the people it serves.

Reforming India's statistical system isn't just a technical exercise; it's a critical investment in the nation's future. Reliable data is the bedrock of a modern economy and a functioning democracy. By making our statistical system more credible, transparent, and responsive, we can empower policymakers to make better decisions, attract more investment, and ensure that India's growth is truly inclusive and sustainable.