When random disturbances are regularly introduced into a dynamical system over time, its small-signal stability is determined by the energy of perturbations accumulated in the system. To analyze this perturbation energy, this paper proposes a novel physically motivated Lyapunov modal analysis (LMA) framework, which combines selective modal analysis with the spectral decompositions of specially chosen Lyapunov functions. This approach allows the modal interactions in dynamical systems to be characterized and estimated in connection with specific state variables. Conventional participation factors characterize the relative contribution of the system modes and state variables to the evolution of states and modes, respectively. In contrast, the proposed Lyapunov participation factors characterize similar contributions to corresponding Lyapunov functions, which determine the integral energy associated with the states and modes on an infinite or finite time interval. This allows the estimation of modal interactions in terms of total energy produced by their mutual actions over time. Using a two-area four-machines power system, we demonstrate that LMA reliably identifies resonant modal interactions, merging of modes, and loss of stability, even for a linear model, and associates them with certain state variables. The Lyapunov participation factors corresponding to the selected part of the system spectrum can be calculated independently and serve as a basis for rapid real-time calculations of critical mode behaviors in large-scale dynamical systems.