Indian Statistical System - Report
Revamping India's statistical system is crucial for its economic
future. The system, once a source of global repute, requires changes now to
keep up with the emerging scenario . To navigate the country's economic and
social challenges, we need to urgently invest in a modern, credible, and
independent statistical framework.
The current system suffers from a set of challenges. Persistent
delays and leave policymakers and businesses operating in the dark. The lack of
disaggregated data at the state and district levels means that
one-size-fits-all policies are made for a diverse country with vast regional
disparities.
The Indian statistical system faces a wide array of challenges
that hinder its ability to provide accurate, timely, and reliable data for
effective governance and informed public discourse. These issues range from
fundamental problems in data collection and methodology to systemic and
institutional weaknesses.
Data
Inconsistency and Credibility Issues
The credibility of India's statistical system has been
significantly eroded by frequent revisions of key economic indicators, such as
GDP growth rates, and noticeable discrepancies between data from different
government sources. This lack of a single, coherent narrative undermines public
and investor trust in the official data.
Delays in
Data Collection and Dissemination
A major challenge is the considerable time lag between data
collection and release. Critical surveys, like those on employment and
household consumption, are often published years after the data is collected,
making the information stale and less effective for addressing current policy
needs.
Outdated
Methodologies
India's statistical system still relies heavily on traditional,
manual survey methods, which were designed for a different era. These
methodologies struggle to accurately capture the complexities of a modern,
digitized economy, including the rise of the service sector, e-commerce, and
the gig economy.
Lack of
Real-time Data
For a rapidly evolving economy, the absence of high-frequency,
real-time data is a significant handicap. Policymakers need up-to-the-minute
information on key economic and social indicators to respond quickly to dynamic
issues like inflation spikes or unemployment trends.
Weak
Institutional Autonomy
The National Statistical Commission (NSC) and the National
Statistical Office (NSO) lack the full autonomy necessary to operate free from external
pressures.
Inadequate
Coordination
Poor coordination among central ministries, state statistical
bureaus, and various data-collecting agencies results in duplicated efforts and
inconsistent datasets. This fragmentation leads to a disjointed statistical
landscape where different bodies produce conflicting figures on the same
subject.
Insufficient
Funding and Human Resources
Statistical departments are often underfunded and face a severe
shortage of skilled personnel. This lack of investment hampers their ability to
conduct large-scale, comprehensive surveys and adopt modern technologies,
further perpetuating the system's reliance on outdated methods.
Poor Data
Granularity
Many surveys provide only broad national aggregates, which are
insufficient for targeted policy interventions. There is a critical need for
more granular, disaggregated data that can shed light on regional disparities,
gender inequalities, and caste-based differences at the district or
sub-district level.
Challenges in
Measuring the Informal Sector
India's economy has a massive informal sector, which is very
difficult to measure. The lack of accurate data on informal enterprises and
employment leads to flawed estimates of GDP, labour force participation, and
overall economic activity, presenting a distorted picture of the economy.
Lack of a
Legal Framework
The absence of a robust legal framework for official statistics is
a major systemic weakness. This leaves the system without a clear, legally
binding mandate for data collection and dissemination, and without clear policy
guidelines, hindering accountability and transparency.
Poor Data
Quality
The quality of administrative data, which is generated by various
government departments as part of their routine functions, is often poor. This
data is not standardized and contains significant errors and inconsistencies,
making it difficult to integrate into the official statistical system for use.
Gaps in Data
Coverage
Significant data gaps exist in critical sectors like employment,
health, and social indicators. For example, India has not released official
poverty estimates since the 2011-12 consumption survey, leaving a decade-long
vacuum in a crucial area of public policy.
Technological
Lag
The Indian statistical system has been slow to adopt modern
technologies like big data analytics, AI, machine learning, and cloud
computing. This technological lag prevents it from processing large datasets
efficiently and from moving beyond traditional survey methods.
Sub-optimal
Use of Administrative Data
India generates vast amounts of administrative data from schemes
like the GST, Aadhaar, and various government portals. However, this data is
not effectively integrated into the official statistical system, representing a
missed opportunity to supplement and validate traditional survey data.
Lack of Data
Transparency
The methodologies used for data collection and the raw data from
surveys are not always made publicly available. This lack of transparency
hinders independent verification, analysis, and research, making it difficult
for academics and experts to scrutinize the official figures.
Inadequate
Training and Capacity Building
The skills of government statisticians often do not keep pace with
global advancements in data science and technology. There is a significant need
for continuous training and capacity-building programs to equip them with the
tools and knowledge needed for a modern statistical system.
Shortage of a
Statistical Cadre
India faces a shortage of a dedicated, professionally trained
cadre of statisticians. Many key positions are filled by general bureaucrats
who may lack the specialized skills and expertise required for rigorous
statistical work, impacting the quality of data collection and analysis.
Poor Data
Dissemination
Official data is often not published in user-friendly formats,
such as APIs or interactive dashboards. It is frequently released in static,
difficult-to-parse formats like PDFs, making it challenging for researchers,
policymakers, and the public to access, analyse, and utilize the information.
Over-reliance
on Estimates
The statistical system often relies on extrapolating data from
small sample sizes to produce national-level estimates. This can introduce
significant inaccuracies and sampling errors, particularly when the sample size
is not representative or is not properly "blown up" to reflect the
diversity and scale of the country.
Action Taken by the Government
Government has initiated several measures to address these
challenges, though progress has been mixed.
Institutional
Reforms
The government has undertaken significant institutional reforms
to streamline the statistical system. A major step was the restructuring of
the Ministry of Statistics and Programme Implementation (MoSPI) and the
creation of the National Statistical Office (NSO) in 2019. The NSO was
formed by merging the two primary bodies: the National Sample Survey Office
(NSSO), which was responsible for large-scale sample surveys, and the Central
Statistical Office (CSO), which handled national accounts and industrial
statistics. This consolidation was intended to improve coordination and avoid
data discrepancies by bringing key functions under a single umbrella.
National
Statistical Commission (NSC)
The government established the National Statistical Commission
(NSC) as an advisory body to the government. The primary goal of the NSC is
to bring a higher degree of professionalism and scientific standards to
the collection, analysis, and dissemination of official statistics. By
providing a platform for independent expert advice, the NSC aims to enhance the
credibility and autonomy of the statistical system. However, its effectiveness
has been debated, with some critics pointing to its limited executive powers
and the fact that its recommendations are not always binding.
Support for Statistical Strengthening (SSS) Scheme
MoSPI launched the Support for Statistical Strengthening (SSS)
Scheme to address the weaknesses in state-level statistical systems. Under
this scheme, the central government provides financial and technical
assistance to state statistical bureaus to improve their infrastructure,
data collection methods, and human resources. The objective is to ensure that
state-level data is collected more uniformly and to a higher standard, which is
crucial for producing accurate national aggregates and for enabling granular,
district-level analysis.
Adoption of
Technology
To improve the timeliness and accuracy of data collection, the
government has pushed for the adoption of technology. This includes the
use of tablets for Computer-Assisted Personal Interviewing (CAPI) in
national surveys. Instead of using traditional paper questionnaires, surveyors
now use digital devices to record responses. This move reduces errors,
accelerates data submission, and allows for near real-time monitoring of survey
progress.
Draft
National Policy on Official Statistics (NPOS)
Recognizing the need for a comprehensive legal framework, MoSPI
has proposed the Draft National Policy on Official Statistics (NPOS).
This policy aims to provide a clear legal and institutional framework
for the statistical system. The NPOS is designed to modernize the system,
ensure data quality, and clarify the roles and responsibilities of various
data-collecting agencies. It also seeks to establish a more transparent and
credible system that can meet the demands of a modern economy.
Utilizing
Administrative Data
The government has made efforts to utilize administrative data
from various government programs and databases. A key example is the ongoing
initiative to integrate data from the Goods and Services Tax Network (GSTN)
into national accounts and economic statistics. By using GSTN data, the
government can get a more accurate and timely picture of formal sector economic
activity, reducing the reliance on traditional, time-consuming surveys. This is
part of a broader push to leverage the vast amounts of digital data being
generated by government schemes.
Release
Calendars
To improve timeliness and predictability, the government
has made it a practice to publish advance release calendars for key
statistical reports. This allows policymakers, researchers, and the public to
know when important data, such as GDP figures or inflation numbers, will be
released. This move enhances transparency and helps to mitigate concerns about
data being withheld or delayed for political reasons.
Strategies for improvement
A comprehensive action plan to revamp the Indian statistical
system can address institutional, technological, and human resource challenges
to restore credibility and provide reliable data for a modern economy.
Strengthen
Institutional Autonomy
A foundational step is to make the National Statistical
Commission (NSC) a permanent, statutory, and fully autonomous body. It could
have the legal authority to audit and certify data quality, ensuring all
official statistics meet consistent standards. By operating independently , the
NSC can ensure the objectivity of data.
Modernize
Data Collection
The system can transition from outdated, paper-based surveys to a
fully digital, real-time data collection system. Using technologies like
Computer-Assisted Personal Interviewing (CAPI) and mobile apps allows
field surveyors to instantly transmit data, drastically reducing time lags and
improving accuracy. This shift is crucial for providing timely insights for
dynamic policymaking.
Establish a
National Data Warehouse
Creating a centralized, secure data warehouse for all
official statistics is vital. This repository would provide a single, reliable
source of information, eliminating inconsistencies and making data easily
accessible to researchers, policymakers, and the public, while ensuring
security and privacy.
Integrate Big
Data and AI
The government can effectively utilize the vast amounts of
administrative data generated by sources like the Goods and Services Tax
Network (GSTN) and the Unified Payments Interface (UPI). By using advanced
analytics and Artificial Intelligence (AI), it can generate more granular
and real-time insights than traditional surveys alone.
Conduct a
Decennial Census
The decennial population census is the foundational
baseline for all official statistics. Prioritizing and conducting the 2021
census without further delay is critical to update demographic and
socio-economic data, which are vital for policymaking, resource allocation, and
providing a reliable framework for all other surveys.
Create a
Statistical Reforms Commission
Constituting a high-level, time-bound Statistical Reforms
Commission is necessary to diagnose deep-rooted systemic issues. This body could
have a clear mandate to recommend a new, robust legal framework to govern the
entire statistical system and restore its integrity.
Overhaul
Human Resources
The system needs a major overhaul of its human resources. This
involves recruiting and training a dedicated, professional Indian
Statistical Service (ISS) cadre and providing continuous skill
development in modern data science, analytics, and data governance. This
ensures the workforce can handle new technologies and complex datasets.
Increase
Funding
Significantly increasing budgetary allocations for
statistical agencies is essential. This funding is crucial for investing in
modern technology, building necessary infrastructure, and attracting and
retaining skilled personnel, all of which are critical for the system's
long-term sustainability.
Improve Data
Granularity
Future surveys can be designed to collect disaggregated data
at the state, district, and sub-district levels. This level of detail is
essential for creating targeted policy interventions that address specific
regional disparities or social issues, rather than relying on broad national
averages.
Bridge Data
Gaps
New surveys can be launched or existing ones revamped to collect timely
and reliable data on key indicators where significant gaps exist,
especially for employment and household consumption. This would provide
policymakers with up-to-date information on the economy and social welfare.
Enhance Data
Transparency
To rebuild trust, it's vital to mandate the public release of
all anonymized raw data and detailed survey methodologies. This
transparency would enable independent scrutiny and academic research, allowing
for data validation and new analysis.
Standardize
Data Protocols
Implementing uniform standards and protocols across all
central ministries and state governments for data collection and storage is
crucial. This would ensure consistency, interoperability, and the ability to
compare data from different sources without discrepancies.
Strengthen
State Statistical Systems
The central government can provide additional funds and
technical support to state-level statistical bureaus. Strengthening these
systems is crucial because much of the ground-level data is collected at the
state and local levels, impacting the quality of national aggregates.
Develop a
National Statistical Policy
Enacting a new, legally binding National Policy on Official
Statistics is a foundational step. This policy would provide a
comprehensive legal framework to govern all aspects of the statistical system,
from data collection and quality standards to dissemination protocols and
accountability.
Engage with
Stakeholders
Creating a formal mechanism for continuous feedback from data
users, including economists, researchers, the private sector, and civil
society, is essential. This ensures the system produces data that is relevant
and useful to a wide range of users.
Establish a
Data Quality Assurance System
Implementing a rigorous, third-party audit system is
necessary to verify the quality and reliability of all official statistics. An
independent body could perform regular checks to ensure data accuracy and
adherence to best practices, enhancing the system’s credibility.
Measure the
Informal Sector
Developing innovative and robust methodologies is critical to accurately
measure economic activity in the informal and unorganized sectors. This
could involve using a combination of survey data, administrative records, and
new technologies to capture this hard-to-measure part of the economy.
Modernize
National Accounts
The methodology for calculating key macroeconomic indicators like
GDP needs to be updated to align with international standards and
reflect the modernizing Indian economy, including the digital and gig
economies, for a more accurate representation of economic activity.
Launch a
Public Awareness Campaign
A public awareness campaign could be launched to educate people
on the importance of statistics and the methodology behind them. This would
not only help rebuild public trust but also encourage greater participation in
surveys, leading to better data quality.
Foster
International Collaboration
Collaborating with international statistical organizations
and academic institutions can help India adopt global best practices, learn
from other countries' experiences, and modernize its methodologies. This will
ensure that India's statistical system is on par with global standards.
Comparison of the statistical systems of India, the US, the UK,
and Japan
Structure
& Organization
India: Has a
decentralized statistical system. The
Ministry of Statistics and Programme Implementation (MoSPI) is the nodal
agency, and it contains the National Statistical Office (NSO), which
consists of the Central Statistical Office (CSO) and the National Sample Survey
Office (NSSO). State-level
Directorates of Economics & Statistics (DES) exist for coordination.
US: Highly
decentralized. Over 100
agencies are involved, with 13 Principal Statistical Agencies (PSAs)
responsible for the bulk of official data. Coordination is managed by the Office of Management and Budget
(OMB) and the Chief Statistician of the United States.
UK: A mix of
centralized and decentralized. The UK Statistics Authority (UKSA)
oversees the system, with the Office for National Statistics (ONS) as
the main producer of statistics. Other government departments also produce statistics.
Japan:
Decentralized.
While the Statistics Bureau of Japan (SBJ) under the Ministry of
Internal Affairs and Communications (MIC) conducts fundamental surveys, various
other ministries have their own statistical departments.
Legal
Framework
India: Governed by
the Collection of Statistics Act, 2008, which allows the government to
collect statistics on a wide range of subjects. Other specific acts, like the Census
Act, 1948, also exist.
US: Has multiple
laws. The Confidential Information Protection and Statistical Efficiency Act
(CIPSEA) of 2018 is a key piece of legislation that ensures confidentiality
and allows for data sharing among statistical agencies.
UK: The Statistics
and Registration Service Act 2007 established the UKSA and its legal
authority to produce and regulate statistics.
Japan: The Statistics
Act (2007) provides the legal basis for the production of official
statistics and outlines the roles of various statistical bodies.
Key Nodal
Agency
India: Ministry of
Statistics and Programme Implementation (MoSPI).
US: Office of
Management and Budget (OMB), specifically the Office of Information and
Regulatory Affairs (OIRA).
UK: UK
Statistics Authority (UKSA).
Japan: Statistics
Bureau of Japan (SBJ) under the Ministry of Internal Affairs and Communications
(MIC).
Primary Data
Producers
India: The National
Statistical Office (NSO), which includes the Central Statistical Office (CSO)
and the National Sample Survey Office (NSSO).
US: The 13
Principal Statistical Agencies, including the Bureau of Economic Analysis
(BEA), Bureau of Labor Statistics (BLS), and the Census Bureau.
UK: The Office
for National Statistics (ONS) is the largest producer, with other government
departments also producing statistics.
Japan: The
Statistics Bureau of Japan (SBJ) and statistical departments within other
ministries.
Role of a
Chief Statistician
India: The
Secretary of MoSPI holds a similar role but lacks the statutory independence of
counterparts in other countries.
US: The Chief
Statistician of the United States in the OMB provides oversight and
coordination.
UK: The National
Statistician is the Head of the Government Statistical Service (GSS) and is
the Chief Executive of the ONS.
Japan: The
Director-General of the Statistics Bureau serves as the head of Japan's
statistical system.
Coordination
India: Coordination
happens through various committees and the NSO, but it can be challenging due
to the decentralized structure and limited statutory power of the National
Statistical Commission (NSC).
US: Coordinated
by the OMB through interagency councils like the Interagency Council on
Statistical Policy (ICSP).
UK: Coordination
is facilitated by the UKSA and the Government Statistical Service (GSS),
a network of all official statisticians.
Japan: Coordination
is managed by the Statistics Commission, which provides advice and oversight to
the MIC.
Statistical
Independence
India: The
statistical system has faced concerns about its autonomy and potential
interference in data releases. The National Statistical Commission (NSC) is an
advisory body but lacks statutory authority to enforce its recommendations.
US: The
decentralized structure and the Evidence-Based Policymaking Act of 2018
provide a high degree of independence. Agencies are legally mandated to produce
and disseminate statistics without political interference.
UK: The UKSA is
an independent body that ensures the production of impartial, high-quality
statistics, promoting public trust. The Code of Practice for Statistics
is a key tool for ensuring integrity.
Japan: The
Statistics Act and a well-established civil service tradition provide a strong
framework for statistical independence.
Quality
Assurance & Standards
India: The National
Statistical Commission (NSC) sets standards, but their enforcement principles
vary.
US: Standards
and guidelines are set by the OMB and enforced across federal statistical
agencies.
UK: The Office
for Statistics Regulation (OSR), the regulatory arm of the UKSA, assesses
official statistics against the Code of Practice for Statistics.
Japan: The
Statistics Commission oversees the quality and methodology of statistical
surveys.
Major
Statistical Outputs
India: Key outputs
include the National Accounts Statistics (NAS), Index of Industrial
Production (IIP), Consumer Price Index (CPI), and results from the Annual
Survey of Industries (ASI) and various socio-economic surveys.
US: Major
outputs include the Gross Domestic Product (GDP), Consumer Price
Index (CPI), Unemployment Rate, and the Current Population Survey
(CPS).
UK: Key outputs
include the Consumer Price Index (CPI), GDP, Labour Market
Statistics, and the UK Census.
Japan: Major
outputs include the Consumer Price Index (CPI), Labour Force Survey,
and the Population Census.
Data
Accessibility
India: Data is
available on the MoSPI website and various ministry portals, but it can
sometimes be fragmented and lack comprehensive metadata.
US: Data is
highly accessible through various agency websites and a central portal, data.gov.
There is a strong emphasis on providing microdata for research.
UK: Data is
readily available on the ONS website, with many datasets open to the public.
There are also secure data research environments for more sensitive data.
Japan: Data is
provided through the Portal Site of Official Statistics of Japan (e-Stat),
which offers a consolidated platform.
Data
Confidentiality
India: The
Collection of Statistics Act provides for data confidentiality, with strong
penalties for misuse.
US: The CIPSEA
and other laws provide robust protections for confidential information
collected by statistical agencies.
UK: Statistical
agencies adhere to strict data protection and privacy laws, including the General
Data Protection Regulation (GDPR).
Japan: The
Statistics Act ensures the confidentiality of information collected for
statistical purposes.
Use of
Administrative Data
India: Efforts are
being made to increase the use of administrative data, but the integration and
quality remain a challenge.
US: Statistical
agencies are increasingly using administrative data to reduce survey costs and
respondent burden.
UK: The use of
administrative data is a key part of the statistical system's strategy to
produce more timely and efficient statistics.
Japan:
Administrative data is used, but a heavy reliance on traditional surveys
remains.
Census &
Surveys
India: The Population
Census is conducted every 10 years by the Office of the Registrar General
and Census Commissioner. The NSSO conducts large-scale household and
enterprise surveys.
US: The Census
Bureau conducts the decennial census and other large-scale surveys like the
American Community Survey (ACS).
UK: The ONS
conducts the UK Census every 10 years.
Japan: The SBJ is
responsible for the Population Census, conducted every five years, and
other major surveys.
Human
Resources
India: Has a
dedicated Indian Statistical Service (ISS) cadre for statisticians.
US: Relies on a
combination of federal employees, contractors, and academic professionals.
UK: Has a Government
Statistical Service (GSS), a professional body for all statisticians
working in government.
Japan: The
statistical system employs a dedicated workforce of statisticians and is
supported by a robust training institute.
International
Collaboration
India: Collaborates
with international organizations and participates in global initiatives, but
this could be strengthened.
US: A major
player in international statistical collaboration, often setting global
standards and providing technical assistance.
UK: Actively
involved in international bodies and initiatives, sharing best practices and
data.
Japan: A
significant contributor to international statistical cooperation, especially in
Asia.
The Path
Forward
The government has initiated some reforms, but further reforms are
being implemented. The nation can prioritize a legally binding National
Policy on Official Statistics that provides a clear roadmap for reform. We can
also launch a public awareness campaign to educate citizens on the
importance of statistics and encourage their participation, helping to rebuild
the bridge of trust between the system and the people it serves.
Reforming India's statistical system isn't just a technical
exercise; it's a critical investment in the nation's future. Reliable data is
the bedrock of a modern economy and a functioning democracy. By making our
statistical system more credible, transparent, and responsive, we can empower
policymakers to make better decisions, attract more investment, and ensure that
India's growth is truly inclusive and sustainable.
