Challenging policy problems and what researchers can do about them by Raj Patel

Image of crowded street
Photo credit: Anthony Cullen for Understanding Society

Many social policies can be difficult to formulate and implement well because they depend on a level of knowledge, mobilisation, governance and policy control that is rarely possible in real life. For more complex problems, it may not entirely be clear what is driving a problem, and feedback loops make it difficult to disentangle cause and effect. As societies have become more heterogenous and diverse, with people’s lives and choices flanked by many different factors, policies aimed at the ‘average’ person tend not to work well.

This complexity and diversity raise challenges for the future direction of research – policy interactions. However, the UK is home to a rich set of longitudinal and cohort studies and a world leader in funding such studies. Such research has the capacity to address the policy issues of: siloed working; developing preventative policies; and short termism in decision making. As a result, they have an important role to play in tackling complex social policy issues.

Cross-cutting policies

The first is the challenge of tackling cross-cutting issues and breaking down boundaries between different disciplines and policy areas. This is not new. Social or business problems are rarely confined to a single research discipline – or a single government department.

The Nurse Review (2015) called for more emphasis on multi-disciplinary research, and a recent survey of civil servants on how they could deliver better public services found that the top issue was cross-team collaboration. It is not simply political or policy differences between teams or department that can be a barrier. Given the accent on policy delivery, when it comes to service delivery making sure each part of government’s ‘supply chain’ is not overwhelmed by policy initiatives is also critical.

There is also a culture of ‘working in silos’ with data, identified in the government’s recently published National Data Strategy, designed to further boost better use of data across businesses, government, civil society and individuals. UKRI’s roadmap on research and development infrastructure recognises the potential to unlock value through data linkage.

Public (and private) administrative data offer the prospect of large amounts of information at relatively low cost. These data are created when people interact with public services and can be linked with rich survey data.

Prevention policy

There is ample evidence to show the scarring effects of life events such as adverse childhood experiences, persistent low income, family break-down or mental ill health. Policy interventions can feel like ‘whack-a-mole’, where a problem addressed in one area pops up elsewhere in a different guise at a later date.

A primary function of government is to reduce risks faced by society. The idea of early action to prevent or mitigate harm has been gaining traction, but progress has been slow, particularly in tackling long-standing inequalities. One reason is that preventive discourses on domestic policy tend to dominate in only limited areas: crime, unemployment and worklessness, family problems, particular aspects of health, and early years, for example.

With prevention, action is usually long-term, more complex to control, less visible, harder to measure and more likely to involve controversial choices, making it more difficult to make progress without building coalitions. Some actions to reduce social risks may be politically opposed by those who regard it as ‘social engineering’.

COVID-19, as well as climate change, an ageing society and the growing interest in wellbeing may start to concentrate people’s minds on prevention in a way that opens up new impact opportunities. It is possible to balance immediate needs with longer-term investment – but this will also require research which provides business cases based on better modelling and cost-benefit analysis.

Short-termism in decision making

Then, there is the related but distinct issue of short-termism in policymaking. Policymakers usually make decisions in a complex environment with limited time for reflection. Ministers want to demonstrate progress quickly and are usually rewarded for spending public funds on today’s visible problems rather than reducing future risks. Short-term and symbolic policies will be the right approach for particular situations, but good government can’t simply be about better firefighting – it needs to learn to prevent fires.

Social mobility, poverty, public health, climate change, housing, social care and regional disparities are long-term challenges which have to be addressed across parliaments. The authors of the Institute for Government’s Making policy stick report say that staying focused on long-term policy goals has been a weakness of government. They identify opportunities, pointing to the benefits of putting ‘strategy capability’ at the heart of government with a ‘licence to operate’.

Stepping up – what can longitudinal studies offer

Longitudinal studies play an important role in research and policy because of the unique data they offer. They collect structured information on children, adults and households, often across multiple generations, and over many years and decades. Their explanatory and predictive power comes from their design, tracking the same people over time, rather than providing a snapshot of people’s lives at a point in time.

They offer greater scope to understand the causes and consequences of change on individuals, families and groups – with policy applications, for example, in early learning, breast feeding, tackling poverty and worklessness, epidemiology and health programmes, savings and pensions.

Longitudinal studies face many common challenges with other types of social science research when it comes to knowledge exchange and policy impact, but could they have a greater impact given their properties? These are explored in a new report, Transforming social policies: insights, ideas and challenges for mobilising data and evidence.

New ways of seeing and understanding will require more integrated data, but the challenge is access, quality assurance, linkage, extracting value from the data, and maintaining a high ethical and privacy threshold. On breaking down silos, there have been a few break throughs on data linkage across fields such as education attainment, hospital episodes and primary care, with other linkage projects in the pipeline – but a long way to go.

In the context of ‘building back better’ from the pandemic, longitudinal research can play a bigger role in moving the dial, but this will require more creativity in how researchers go about generating impact. The Transforming Social Policies report, published by Understanding Society, aims to help readers understand the policy process, and proposes new forms of collaboration: to change the way social problems are thought about; support bottom-up social innovation; and strengthen the business case for early action to pre-empt welfare needs.


Raj Patel (@RajPatel08) is the Associate Director of Policy at Understanding Society, Institute for Social and Economic Research, University of Essex. This blog was originally written for the LSE Impact Blog.

Disclaimer: All our blogs are the views of the author(s) and not the SPA.