In the zebra finch, singing behavior is driven by a sequence of bursts within premotor neurons located in the forebrain nucleus HVC (proper name). In addition to these excitatory projection neurons, HVC also contains inhibitory interneurons with a role in premotor patterning that is unclear. Here, we used a range of electrophysiological and behavioral observations to test previously described models suggesting discrete functional roles for inhibitory interneurons in song production. We show that single HVC premotor neuron bursts are sufficient to drive structured activity within the interneuron network because of pervasive and facilitating synaptic connections. We characterize interneuron activity during singing and describe reliable pauses in the firing of those neurons. We then demonstrate that these gaps in inhibition are likely to be necessary for driving normal bursting behavior in HVC premotor neurons and suggest that structured inhibition and excitation may be a general mechanism enabling sequence generation in other circuits.