This paper develops adaptive approach to the controlling observable Markov chains with a finite number of states. We apply Robust Mirror Descent Randomized Control Algorithm (RMDRCA) to a class of homogeneous finite Markov chains governed by the multi-armed bandit with unknown mean losses. It develops approach represented in [Nazin and Miller (2011a)] and [Nazin and Miller (2011b)]. As opposed to the partially observable Markov decision process an adaptive approach does not presuppose the knowledge of probabilistic characteristics of random perturbations and permits to obtain the control strategy with known rate of convergence to the optimal solution. We propose the concrete RMDRCA and prove the explicit, non-asymptotic upper bound for the mean losses at any current time. Numerical example illustrates theoretical results.