We study the influence of mean-field cubic nonlinearity on Aharonov-Bohm caging in a diamond lattice with synthetic magnetic flux. For sufficiently weak nonlinearities, the Aharonov-Bohm caging persists as periodic nonlinear breathing dynamics. Above a critical nonlinearity, symmetry breaking induces a sharp transition in the dynamics and enables stronger wave-packet spreading. This transition is distinct from other flatband networks, where continuous spreading is induced by effective nonlinear hopping or resonances with delocalized modes and is in contrast to the quantum limit, where two-particle hopping enables arbitrarily large spreading. This nonlinear symmetry-breaking transition is readily observable in femtosecond laser-written waveguide arrays.