Mineralized collagen fibrils are highly conserved nanostructural building blocks of bone. By a combination of molecular dynamics simulation and theoretical analysis it is shown that the characteristic nanostructure of mineralized collagen fibrils is vital for its high strength and its ability to sustain large deformation, as is relevant to the physiological role of bone, creating a strong and tough material. An analysis of the molecular mechanisms of protein and mineral phases under large deformation of mineralized collagen fibrils reveals a fibrillar toughening mechanism that leads to a manifold increase of energy dissipation compared to fibrils without mineral phase. This fibrillar toughening mechanism increases the resistance to fracture by forming large local yield regions around crack-like defects, a mechanism that protects the integrity of the entire structure by allowing for localized failure. As a consequence, mineralized collagen fibrils are able to tolerate microcracks of the order of several hundred micrometres in size without causing any macroscopic failure of the tissue, which may be essential to enable bone remodelling. The analysis proves that adding nanoscopic small platelets to collagen fibrils increases their Young's modulus and yield strength as well as their fracture strength. We find that mineralized collagen fibrils have a Young's modulus of 6.23 GPa (versus 4.59 GPa for the collagen fibril), yield at a tensile strain of 6.7% (versus 5% for the collagen fibril) and feature a fracture stress of 0.6 GPa (versus 0.3 GPa for the collagen fibril).