The networks of threshold elements are considered. The notion of a stationary ensemble is introduced. We set out necessary and sufficient conditions for a network to be an ensemble. We show that for two ensembles with common elements switching on one ensemble does not necessarily lead to switching on the other. A representation of an ensemble as a finite state machine is proposed. We then show how this representation can help study the processes of switching ensembles on and off. We note that a network of ensembles may be interpreted in neurobiological terms as a basis for a long-term memory model. In social sciences it may serve as a network version of the collective social threshold behavior model.