Discovery of protein-binding fragments for precisely defined binding sites is an unmet challenge so far. Here, we investigate formylglycine as a molecular probe for the sensitive detection of fragments binding to a spatially defined protein site. Formylglycine peptide 3 was derived from a phosphotyrosine-containing peptide substrate of protein tyrosine phosphatase PTP1B by replacing the phosphorylated amino acid with the reactive electrophile. Fragment ligation with formylglycine occurred in-situ in aqueous physiological buffer. Structures and kinetics were validated by NMR spectroscopy. Screening and hit validation revealed fluorinated and non-fluorinated hit fragments being able to replace the native hosphotyrosine residue. The formylglycine probe identified low-affinity fragments with high spatial resolution as substantiated by molecular modelling. The best fragment hit, 4-amino-phenyl-acetic acid, was converted into a cellularly active, nanomolar inhibitor of the protein tyrosine phosphatase SHP2.