SALAHUDDIN

The proliferation of data-driven organizations and the growing acceptance of data-driven interventions in the social sector speaks to an increasing recognition of the value of data in identifying social problems and developing solutions. The private sector has already been successful in leveraging data to improve productivity, increase profits, and gain a competitive edge. However, historically, the public and nonprofit sectors have been slower to adopt data-driven approaches.

In recent years, this has begun to change. Policymakers and development sector professionals are increasingly using data to identify areas where social interventions are needed and to design more cost-effective and efficient solutions. Rigorous statistical techniques, such as those employed in randomised controlled trials, have invoked a deep interest in social researchers. This specialized approach brings a sense of scientific rigor to the realm of social studies, enabling researchers to isolate causality and draw robust conclusions about the effectiveness of interventions. This trend has the potential to transform the way social problems are addressed, as traditional approaches have often been ineffective. In Pakistan, several think tanks have emerged with the goal of conducting high quality data-driven research. However, it is yet to see if these institutions can produce useful research and insights that can positively influence policy decisions concerning some of the most pressing issues such as poverty and inequality.

The increasing embrace of data-driven approaches in the social sector holds great promise for driving sustainable change. However, there are several challenges to implementing such strategies, including a lack of strategic urgency. Data is often very limited and exists in silos with no clear direction for its utility. Where data does exist, social organizations lack appropriate skills and personnel to extract insights. Even in instances where data is used extensively, the data teams are often isolated from strategy and policy discussions.

Despite the widespread use of big data in multinational companies, many individuals are frustrated by the nonexistence of data on issues that matter to them. This reality stands out given the vast amount of data being gathered, mined, and analysed by big tech companies. Perhaps the issue is not a lack of data, but the absence of priority and incentives around certain data points that are not captured because they do not provide sufficient economic benefits. Social issues like poverty and inequality are not on the priority lists of multinationals. The explosion in data has not added proportionate benefits to the social sector where social scientists are still scrambling with outdated datasets.

Young data scientists for social impact

The growing demand for data analysts and scientists has resulted from the vast amounts of data being generated by big businesses. According to the US Bureau of Labor Statistics, computer and information scientists’ employment is estimated to grow by 15% from 2019 to 2029, a much faster pace than the average for all occupations. Numerous online courses, such as Harvard University's Big Data for Social Good Certification, and tutorials are available on data science, with YouTube and other social media platforms hosting countless data nerds. Communities run by data enthusiasts, such as Kaggle and Stack Overflow, have facilitated the exchange of knowledge and skills. ChatGPT has has made it very convenient for data analysts to build and optimize their code and explore data with a few prompts.

With the immense volume of data being produced by large enterprises, particularly big tech, it is no surprise that young data scientists and analysts aspire for a career with these businesses, where they get to work on expansive and complex datasets. They also get to enjoy lucrative perks and high salaries. While there are certainly benefits to working in the tech industry, there is also a growing movement of data analysts who are using their skills to drive positive social change. These analysts seek to apply their expertise to big-picture issues such as poverty, education, and healthcare, with the goal of improving outcomes for neglected and vulnerable communities.

One of the biggest challenges facing social impact data analysts is the lack of standardization and quality control in the data they are working with. Unlike the carefully collected and curated datasets used by large enterprises, real-world data used in the social sector is often scattered across a variety of sources and is frequently incomplete, unstructured, or unreliable. Data is often collected using unsophisticated means with no connection to a real-time centralized database. Data analysts working in the social sector must have the skills to harmonize and clean the data before they can extract any useful insights. It is good practice to throw yourself into a lake of messy data and carve your way out. Regrettably, for many of us, opportunities to engage with genuine real-world data are limited beyond our professional spheres.

Another major challenge for social impact data analysts is the lack of resources available to support their work. In the tech industry, companies have invested heavily in data infrastructure, allowing analysts to access large datasets and use sophisticated tools for analysis and visualization. In contrast, social impact organizations often lack the resources necessary to invest in data systems and tools, leaving analysts to work with limited resources and outdated technology. Like many individuals, I also truly believed that I could use my data science skills for social good. Over the course of my professional career however, I struggled to find ways in which I could use my data expertise in influencing policy decisions and driving social change. I was often frustrated by a lack of urgency and priority regarding data usage. Even initiating a conversation around data seemed futile at times.

Despite these challenges, there are many compelling reasons for data analysts to pursue a career in social impact. By cultivating a culture of data-mindedness and using their skills to uncover useful insights, analysts can contribute to some of the most pressing social issues of our time. Additionally, working in the social sector can offer a sense of purpose and fulfillment that is difficult to find in other industries. The social sector is currently in desperate need of a meticulous, evidence-driven approach that can only be furnished by skilled data experts. The convergence of data expertise with the social sector presents a symbiotic relationship, wherein the analytical prowess of these experts can be channeled towards fostering positive change. The social sector should priotitize data and create space for data experts who can help us better guage the impact from social interventions.

Mistakes to steer clear of

As the field of social impact data analysis continues to grow and evolve, we will hopefully see more data analysts pursuing careers in this area. By working together to overcome the challenges of working with messy, unstructured data, these analysts can discern some valuable insights that can help us build a more just and equitable world for all. However, the emphasis on data-driven interventions should not be used as an excuse to ignore the subjective experiences of individuals. An over-reliance on quantitative data can miss individual life experiences and social perspectives which are difficult to capture in numbers. It is essential to strike a balance between data-driven insights and an appreciation for the richness of subjective human experiences which can be captured, even if imprecisely, through qualitative methods. Incorporating the subjective into the data-driven paradigm requires a conscious effort to complement quantitative insights with qualitative understanding. This involves engaging in empathetic conversations, conducting in-depth interviews, and actively listening to individual stories.

The domain knowledge is essential when it comes to the social sector. One must practice caution when employing data that has the potential to influence outcomes for socially vulnerable populations. Mathematical models and algorithms are not necessarily neutral in nature. They are influenced by the data they are trained on, which can reflect existing biases and disparities. The notion of neutrality crumbles in the face of such realities. Without necessary guardrails, integrity of the data, and data privacy policies, these models could lead to harmful outcomes and reinforce inequality. This theme has been discussed extensively by Cathy O'Neil in her book Weapons of Math Destruction. A model's output is as good as the data that goes into it. Therefore, it is indispensable to understand the context and the consequences of the data before applying it to social issues.

Good quality data holds as much potential for the social sector as it does for the private sector. Why do we spend all our efforts and resources in improving 'customer service' or the 'user experience' while ignoring and spending an abysmal amount on improving social issues? This disproportionate focus leads to the widening of the gap between communities and groups. If we treat social issues the same way that the for-profit sector treats customer loyalty and experience, then we would be able to contain the long-term adverse effects of social issues. The returns from investing in data for creating social impact are significantly more extensive and profound compared to those for purely profit-making enterprises.