Structural alterations in oligosaccharide structure are often associated with disease, positioning clinical glycomics an emerging tool for diagnostics. This is most commonly achieved using a controlled collision-induced dissociation (CID) of larger oligosaccharides into fragments and measuring their mass in a mass spectrometer. Due to the complexity of oligosaccharides, and particularly their unusual fragmentation mechanisms, the underlying processes are poorly understood. Deciphering glycan fragmentation and making it understandable is highly desirable and would transform the field of glycomics from an expert technique into a widely applicable tool available to non-specialists. Here, we review the current knowledge of glycan fragmentation mechanisms in CID, with particular emphasis on hexose migrations and the anomeric memory. We discuss challenges and perspectives for future investigations, opening the window to widespread use of glycomics in clinical applications based on a fundamental understanding of glycan fragmentation.