In this paper we explore how ‘ability’ is currently conceptualised in physical education and with what effects for different groups of young people. We interrogate approaches to theorizing ability in physical education that draw on sociological and phenomenological ‘foundations’ together with notions of ability as ‘physical’ and ‘cultural capital’ drawn from the work of Bourdieu. We also look to data we and others have collected across a number of empirical projects to ask: where do we find talk about what we might identify as ‘ability’ in the context of physical education and sport; how is it talked about? and in what ways might this further our thinking of the meaning of ‘ability’ in physical education and school based sport? Our findings suggest that physical ability is far from a neutral concept and that how it is understood has important consequences for young people in relation to gender, race and social class. We argue that ongoing discussions around what we mean by ability, how we use it, and in relation to whom, are crucial in physical education where organized sport, recreation and exercise remain privileged over other constituents of physical culture.