We discuss the physical properties of realistic memristive, memcapacitive and meminductive systems. In particular, by employing the well-known theory of response functions and microscopic derivations, we show that resistors, capacitors and inductors with memory emerge naturally in the response of systems-especially those of nanoscale dimensions-subjected to external perturbations. As a consequence, since memristances, memcapacitances and meminductances are simply response functions, they are not necessarily finite. This means that, unlike what has always been argued in some literature, diverging and non-crossing input-output curves of all these memory elements are physically possible in both quantum and classical regimes. For similar reasons, it is not surprising to find memcapacitances and meminductances that acquire negative values at certain times during dynamics, while the passivity criterion of memristive systems imposes always a non-negative value on the resistance at any given time. We finally show that ideal memristors, namely those whose state depends only on the charge that flows through them (or on the history of the voltage), are subject to very strict physical conditions and are unable to protect their memory state against the unavoidable fluctuations, and therefore are susceptible to a stochastic catastrophe. Similar considerations apply to ideal memcapacitors and meminductors.