ESC is the escape character, M-x, that actually generates two key events, first the ESC event, second the x. (actually, that's only true in terminal mode since in X M-x generates a special event, but this event is mapped to ESC x). The problem is that in terminal mode the escape key does not generate ESC.
That said, in order to have the escape key working correctly in terminal, Evil has to catch the ESC event and possibly translate it to M-x or leaving insert state with ESC would not work). In other words, the event ESC typically never reaches those keymaps because it is translated to evil-intercept-esc can be used to customize this behavior (but only in newly created frames).