That’s what I tell all my students from the beginning and it is really important for us as ALTs to do that. After some time, they may realise that you seem to understand Japanese somewhat and it is then you can and should say you only know a little. And of course all this while saying all that in English. I didn’t see the benefits of pretending to not speak Japanese initially but I came to understand that after some time and while it may be difficult at first, once you get used to it, you’ll even find it more natural to speak English with them than Japanese.
Unfortunately, many ALTs are unable to resist the temptation to display or make known their ability to speak the language that they actually tell students they do understand and/or speak Japanese. If students know you have a certain level of grasp of the language, you would have officially rendered your existence at school null. By giving students no reason to speak to you in English, you are practically not much different from the Japanese teachers of English and that would bring up the question of why the school should have you there at all. But you would realise that at times, when they really have no idea how to convey what they wanna say to you in English, you can listen to their Japanese question and reply in English. Chance is, they’d realise you understand Japanese somewhat and it’s always good to reiterate that you only know a little. Some of them don’t even realise that you understand Japanese because the reply in English just flowed so naturally.
Like what I’ve mentioned in an earlier post, companies want Japanese-speaking ALTs to communicate with the teachers. Not the students.