We develop an approach to the computer study of a threedimensional nonlinear model of population dynamics, taking into account trophic chains and competition in prey populations. This approach is based on the application of stability research methods, numerical methods for solving ordinary differential equations, and modern methods of parametric optimization. A qualitative study of the system is carried out, the states of equilibrium are found, of the species population dynamics graphs and corresponding phase portraits are constructed. An assessment of the parameters influence on the model stability and on the permanent coexistence of populations is presented. A software package in the Python language is used as a research implementation tool. The problem of the optimal control of the model with phase constraints is formulated. A control quality criterion is proposed and a generalized algorithm for solving the optimal control problem is developed. The results obtained can be used in problems of modeling and forecasting multidimensional ecological systems, as well as optimal control problems.