Task representations consist of different aspects such as the representations of the relevant stimuli, the abstract rules to be applied, and the actions to be performed. To be flexible in our daily lives, we frequently need to switch between some or all aspects of a task. In the present study, we examined whether switching between abstract task rules and switching between response hands is associated with overlapping regions in the posterior lateral frontal cortex and whether switching between these two aspects of a task representation is neurally implemented by distinct functional brain networks. Subjects performed a cue-based task-switching paradigm where the location of the task cue additionally specified the response hand to be used. Overlapping activity for switching between abstract rules versus response hands was present in the inferior frontal junction area of the posterolateral frontal cortex. This region, however, showed very distinct patterns of functional connectivity depending on the content of the switch: Increased functional connectivity with anterior prefrontal, superior frontal, and hippocampal regions was present for abstract rule switching, whereas response hand switching led to increased coupling with motor regions surrounding the central sulcus. These results reveal that a rather general involvement of the posterior lateral frontal cortex in different switching contexts can be further characterized by highly specific functional interactions with other task-relevant regions, depending on the content of the switch.