Musings of a Restless Mind

When we were confronted with 2 plane crashes near airports in the last week, The Media That Be (TMTB) howled that Trump was going to get us all killed because there is a shortage of Air Traffic Controllers. Trump in his usual attack, then counter-attack mode blamed DEI policies for lowering the standards, recruiting obviously unqualified people, and preventing qualified people from taking the jobs on which peoples’ lives depend. He actually quoted from a policy prominently displayed on the FAA’s website that encouraged actively recruiting those with targeted disabilities, including “hearing, vision, missing extremities, partial paralysis, complete paralysis, epilepsy, severe intellectual disability, psychiatric disability and dwarfism.” TMTB checked the website in realtime and found it said exactly that, but complained that it had been in place since 2013, so he must be to blame for not having repealed it during his first term. Still the idea was planted in the heads of listeners sympathetic to Trump that the FAA had been recruiting people with “complete paralysis”, “missing limbs”, and “severe intellectual disability”, and “psychiatric disability” to be air traffic controllers. Of course the press replied that was for all of the FAA and not just air traffic controllers. Still I wonder what jobs are suitable for those completely paralyzed or with psychiatric disability (i.e. crazy, right?) could fill even elsewhere in the FAA.

If you want to research a hot breaking news story, of course, it’s always prudent to wait until things settle down, and facts start to come out, and sure enough, today, I found a lengthy explanation of what had really been going on in FAA recruitment of air traffic controllers. The internet came through for me, and someone created a comprehensive history of DEI and its impact on would-be air traffic controllers. If you’re curious I highly recommend you read all of it.

That is the background that I lured you here with because what I really want to talk about is something I actually know something about—questionnaires. On New Year’s Eve 2013, the FAA announced that anybody who had taken the previous qualifying test, that had proved remarkably predictive of success for air traffic controllers, was just out of luck. Their scores no longer counted. They were rewriting the test to stop the systemic exclusion of underrepresented minorities. Also, in order to even take the test, now you had to fill out a biographical questionnaire which would also be scored. If you didn’t score high enough on that, you were out of consideration. End of story. Witnesses claim that some organization named the National Black Coalition of Federal Aviation Employees could increase a candidate’s chances with that test. The witness attended a seminar where the attendees were shown all the questions and the preferred answers. I hit the link and took the questionnaire, that was used to prescreen applicants from 2014 to 2018 when the FAA lost a lawsuit that claimed the test discriminated against whites.

My interest in this questionnaire quickly became professional rather than political. I’ve been a professional statistician as part of my work-life, and I’ve written questionnaires, so I know something about their design and usage. Everything about this seemed drawn from a Monty Python sketch. First, no one who cares about the results, assigns a questionnaire, on which someone’s future career hangs, and instructs them to fill it out at their leisure in private. Why would a test taker not lie or cheat given the importance placed on the questionnaire? That was head-shaker number one.

From there, things got stranger and stranger. The survey has 62 questions only 28 of which are scored! Now, professional statisticians might occasionally throw in a few questions that aren’t counted, just to get an idea if the results show them to be valid questions on future surveys, but a majority of the questions?! The next eye-opener is that 12 of the 28 questions that actually counted are ones that can be objectively discovered without self-reporting, like your high school GPA. The only reason you would do that is if you don’t actually care if the question is answered truthfully.

The scoring method also was a head scratcher. Some questions had one answer with a value of 15 points where the other 4 choices out of 5 resulted in 0 points. Other questions were scored 1-5 with points awarded for each answer. A few questions were scored up to 10 points for the “correct” answer.

A few questions asked about your grades in certain subjects with the scores of A=5, B=4, C=0, D or lower=1, can’t remember/NA=1. What could they possibly be looking for in giving you more points for a D, F, or irrelevant than for a C? In what alternate universe does that make sense?

One question asks the number of college credit hours you have for Art/Music/Dance/Drama which seems odd, but the scoring is even more odd. You get 0 for not attending college, 5 for not taking any of those classes, but 0 points for 1-6 hours, 4 points for 7-12 hours, and 1 point for 12 or more credit hours. How could that scoring be anything but random? I defy you to make sense out of it.

Similarly your GPA in high school Arithmetic is scored A=5, B=2, C=0, Lower than C=1, Did not have that course=3. There were no arithmetic classes in my high school. There were math classes–algebra, geometry, calculus. You were supposed to already know how to add and subtract and the times tables.

There was also a pair of questions about the high school and college classes in which you received the lowest grade. For high school you got 15 points for scoring lowest in science and nothing for anything else. For college you got 15 points for scoring lowest in History/Political Science. Despite the odd wording and juxtaposition of the questions, and their questionable significance, those 30 points represent fully a sixth of the whole potential score of 179. Oh, and I’m sure my college English professors will be flattered to learn that if your lowest college grade was in English, it scored 0 points on your aptitude for being an air traffic controller, thus allowing them to feel infinitely superior to the History department. Actually dividing by zero is undefined, but, hey they’re English professors, so let them feel smug.

The question that I found most bizarre was “I learned about the opportunity to apply for an Air Traffic Control Specialist (ATCS) job through:”

• A public notice or media advertisement = 5
• A friend or relative = 0
• College recruitment = 3
• Working in some other capacity for the agency = 3
• Some other way = 0

They forgot to ask if you found out about it via graffiti on a bathroom stall.