I think that this is an attempt to stop the “clean key” problem. Security system keypads (especially outdoor ones) tend not to be cleaned, so as time passes, it is easy to spot the dirty keys. Dirty keys are not being pressed and are therefore not in the passcode.
So look for the clean keys and try combinations from there. In a 4 digit (0-9) keypad, knowing the clean numbers drops the possible codes from 9999 to 24 (if my early morning math holds up).
Also helps the issue of someone looking over the shoulder of a valid person. Chances are they are just seeing the position and not the character pressed. So the keyboard changes and you actually need to know the character not just the old position.
If they aren't distinct, you wouldn't have 4 clean buttons, but just 3 - in which case we also know the repeating digit repeats exactly once and we get 12x3 (36) possible combinations. With two clean buttons, it's 6 (if both repeat) + 4 (if only one repeats) = 10 and if there's 1 that's just one, and a terrible password.
I encountered a case of this in college, where there were four clean digits - a tough task to be sure! But fortunately the digits happened to be the same set of digits that comprised the room number. It took two guesses, because there was a twist - the combination was the room number, backwards.
I guess you would be able to count the number of clean keys and thus know both the number of distinct digits and the digits (but not the order nor which digit that's repeated)
1. You have lots of spy-data samples that reveal which physical key is pressed (perhaps they sound different) and the precise timing of those strikes, but you don't know what scrambled numbers were actually being shown. (And it's always the same code.)
2. The trick is that users take longer to press a number when it's displayed far away from its "normal" position, because they had to seek longer to find it.
3. This means you can infer the true numbers based on how quickly or slowly presses happen versus which physical key is struck.
For a simple example, assume a two-digit code where there are nine keys. If the fastest first press is always the top left corner, and the fastest second press is always the middle, we can guess the code is either 15 or a 75, depending on if the user is accustomed to phones or keyboard numpads.
P.S.: On reflection, I could probably have shortened all that by describing it as a "timing attack" [0] except in meat-space.
One mitigation might be to get the user to enter digits at a consistent pace, by forcing a delay between showing the random layout versus accepting a button press. There would need to be some penalty for early presses, to keep lazy users from just tapping the desired button repeatedly until it became active.
This would be an interesting one to integrate into password entry forms... although you'd need to show the randomised keyboard layout on screen.
Or have a keyboard with oled or e-ink keys, like the Optimus Maximus [0] promised to deliver. It's kinda weird that nobody else seems to have picked up on this concept since then. Probably just impractical or too expensive.
I read that its patents expired in 2016; around 2015 there was a concept for an e-ink button keyboard, but that site is now a plain gambling ad. There's also https://www.nemeio.com/ that still works, but its buttons look like sunken screens under plastic domes.
A number of countries use this when giving your pin for a credit card or similar (I've noticed it in both Greece and India).
I can't help but feel like it's less secure than the default layout - I'm quite good at hiding my PIN and typing quickly, but when the positions of the numbers are randomised, I feel like I practically end up saying my PIN out loud as I try and remember it.
Had this at an ATM recently, and it took a couple of tries at my PIN before I looked at the keypad and realized what was going on. One more wrong PIN and I could have lost my card.
