What is the term for the study of how people make decisions in situations where their outcomes depend on the actions of others?

Game theory

Game theory is used in economics, political science, and biology. The most famous example is the 'Prisoner's Dilemma,' which illustrates why cooperation can be difficult.