Well that’s the core of the problem. Either you’re measuring speed on a test or you’re not. If you are, then people with disabilities unfortunately do not pass the test and that’s the way it is. If you are not, then testing some students but not others is unfair.
At the end of the day setting up a system where different students have different criteria for succeeding, automatically incentivizes students to find the easiest criteria for themselves.
Tests usually do measure the speed. And often they should. But the question here is "the speed of what?". And how do you measure the speed without also measuring the speed of something else as an error?
If you just want to measure speed, we should clock the time the student gets up, until they get to the room where the test is, get's out his pen etc. So students get the same time to do all this.
We are now measuring the speed at which the student is able to do the test material including all the preparatory steps. Students who live further away or have slower cars will get worse grade, but we are just measuring speed, aren't we?
That is a deliberately stupid example, but it shows that is important to ask "speed of what?". When doing a physics exam, what do we want to include in our measurement? The time it takes the person to read an write? Or just the raw speed at which physics knowledge can be applied? What is error and what is measurement?
You can see it as measuring based on different criteria. Or you can see it as trying to get rid of sources of errors that may be vastly different for different students.
It would be great if we could reduce the sources of errors down to zero for everyone. But unfortunately humans are very stochastic in nature, so we cannot do this. But then there has to be an acceptable source of measurement error (typical distribution) and an unacceptable source of measurement error (atypical distribution) and to actually measure based on the same criteria, you need to measure differently based on what you believe the error to be.
At the end of the day setting up a system where different students have different criteria for succeeding, automatically incentivizes students to find the easiest criteria for themselves.