The Case of Josh Renaud

Accused of hacking a website of the State of Missouri, a journalist and his newspaper are being charged for criminal activity. The job of reporting done by Josh Renaud was journalism at its best. It was thorough and it was responsibly reported. The State of Missouri and the Department of Education should quickly apologize and thank him for his efforts.

First posted February 8, 2022

Updated

February 12, 2022
February 16, 2022
March 2, 2022

On October 14^th, 2020, the St. Louis Post-Dispatch posted “Missouri teachers’ Social Security numbers at risk on state agency’s website”, written by Josh Renaud. Mr. Renaud had reported on some rather controversial topics in the past, most particularly the events that unfolded in Ferguson, Missouri in 2014. He had also reported extensively on the political remapping of Missouri’s Congressional, state and local districts. He probably had no idea that a mundane report involving a teacher’s certificate would explode into a major court battle over freedom of the press, the nature of open-source coding of web pages, and the massive broadening of the term “hacker” and what are constituted as invasive techniques.

From what is currently being divulged to the public, Mr. Renaud discovered that social security numbers were embedded in the data that was generated from a web site designed to provide verification that a teacher held a legitimate and current teacher’s certificate. The website in question, as of the time of the writing of this article, is down for maintenance. Using common web analysis tools he discovered that the social security numbers of the teachers were embedded in the string of data that is delivered to the inquirer’s web browser. Since he could only look up one teacher at a time, he tried the same on two other teachers before confirming the discovery. According to the Post-Dispatch, Renaud (and his editor) did the ethical thing by reporting the vulnerability to the Department of Education and withholding the posting of the report until the vulnerability could be addressed. This evidently promptly happened. But the State’s response was swift and rather startling. Instead of thanking the Post-Dispatch, it charged Renaud for “hacking” the system.

According to Press Freedom Trackers (PFT), the Washington Post reported that the specific violation in question was “tampering with computer data — a misdemeanor punishable by up to a year in jail and a $2,000 fine — and that another Missouri code allows a civil suit for damages.“ The PFT report went further to report that the case had been referred to the Cole County Prosecutor and the Missouri State Highway Patrol’s Digital Forensic Unit.

As an IT professional with considerable work in IT security, I found the charge rather baffling. My immediate reaction was that this was a terribly embarrassing misunderstanding of how the web works. Dragged into the mud by the governor’s reaction is the Cole County Prosecutor and the State Highway Patrol. Yet let’s give the State of Missouri the benefit of the doubt. They assert that the technique he used was a so-called multi-phased process of capturing the data and decoding it. From the State’s press release,

“Through a multi-step process, a hacker took the records of at least three educators, decoded the HTML source code, and viewed the social security number (SSN) of those specific educators. “

Furthermore, the press release provided an explanation of how and why social security numbers are used. Inquirers can utilize the last four digits of a teacher’s social security number to better isolate individuals with similar names. It was probably in that element of the search code that the entire social security number was inadvertently included in the data stream.

What we do not see in any of the reports I have read is the actual stream of raw data. That is obviously an issue because it is protected data that should not be exposed to the public. The state says it was a “multi-step” process, whatever that means. Yet the general consensus is that the state has embarked on a quixotic campaign that will only discredit the state’s law enforcement, the Department of Education and the governor himself. I will explain why below and how this is a common problem amongst bureaucrats and politicians.

Let’s begin with the FBI. When handed the details of the case, they promised to investigate the matter further but, upon first reading of the evidence, they saw nothing unusual about the techniques and tools used by Josh Renaud to uncover the private data.

As the Post-Dispatch later reported:

…. on Oct. 13, Angie Robinson, cybersecurity specialist for the state, emailed Department of Public Safety Director Sandra Karsten to inform her that she had forwarded emails from the Post-Dispatch to Kyle Storm with the FBI in St. Louis.

“Kyle informed me that after reading the emails from the reporter that this incident is not an actual network intrusion,” she said.

Instead, she wrote, the FBI agent said the state’s database was “misconfigured,” which “allowed open source tools to be used to query data that should not be public.”

Reaction From the Press

Reaction from the press is universal. They think the state is insane. On the ethical level, Renaud and the Post-Dispatch acted responsibly. This was not a Julian Assange type thing of spilling the raw information before the public. There was a discovery of a vulnerability and it was reported to the Department of Education. They did it by the book.

Secondly, Renaud appears to have used commonly available tools that are used to better understand websites. Why journalists would use these type of tools is not the issue. They are there, as I will demonstrate, in Firefox and other browsers. Web page editors, network analysts and security specialists use these tools for all sorts of reasons. The content they analyze is, by intentional design, available to everyone.

The Source Code

Renaud is accused of viewing source code. When Tim Benners-Lee invented the hypertext markup language (HTML), he did so for the purpose of designing a universal code that any computer system could read. The browsers we use are fundamentally designed to read HTML and it is a code that is standardized through governing international bodies.

Read Page Source Option — Right-Click over Page to Present View Page Source

This source code can be read by anyone. The basic raw bones of a web page you view will have the HTML code available for your viewing. It has to. Anyone who has access to the page will need to read the code to present the page.

Below is an example of the source code that is used on Trello (I use it to organize my life). Access any website, right click on the screen and you will see that one of your options is “View Page Source.”

Clicking on “View Page Source” will present the HTML code for that page.

To protect data, the HTML code is designed to interface with other programming languages. For secured data, all the viewer will see is a reference to a script that takes the information they have entered onto the screen and transfers it a database server through the protected script. To repeat, a well designed website will not permit an unauthorized person to view the script or directly access the database.

What about your passwords or any private information that you may enter onto a web page? You will notice that most websites commence with HTTPS, with the “S” indicating the link to the website is encrypted, so whatever you type cannot be read over the wire. Some websites go a step further and encrypt the data you enter on the screen before transferring it to a secured server “inside the firewall.”

At this juncture, there is no evidence that Renaud utilized “uncommon” tools to identify protected servers behind the firewall. He simply viewed the source code.

Web Tools

I utilize Firefox because it is, by and large, one of the most well-designed and secured browsers on the market. I know there are others and I am certain that they are also just as good, if not better in some respects. But I worked as a security instructor for four years, followed as a performance analysts for several years with the US Forest Service. My tool of choice for the performance analysis of websites was none other than Firefox. People would contact me and say “the web server is responding really slow.” One easy technique was to see what the web page was attempting to do.

Embedded in Firefox is a set of “Web Tools”. Clicking the Menu icon in the upper-right corner of the browser will present a set of web tools that you can use to improve a web page. A sample is provided below using my Substack page (see above).

Note there is an option for “Web Developer Tools”. Clicking on that will split the screen. The example below shows some metrics posted from my the Substack site.

Web Tool Sample — Sample of Web Tools from Firefox

As you can see, there are several tools that developers can use to analyze a web page. More than likely, Renaud used such a tool. The Debugger tool is especially helpful for following where your clicks lead you inside the code itself. It is also a good way to detect recurring errors (one source for “slow” web pages). The main tool I used was the Network tab. Web sites typically draw information from several servers. The Network tab provides response-time metrics that help me identify likely suspects for slow web pages. But they also map out the other sites and devices that the web page communicates with.

The Misconfigured Database

Earlier it was stated that the database was “misconfigured.” What exactly is that? We can only speculate, but I can well imagine how it happened. When a visitor to the certification site enters their request, a script channels that information to a database server. The database server then constructs a query based on the user’s request. As noted above, one of the things an inquirer can use to identify a teacher is the last four digits of their social security number, something that school administrators would have in their possession for anyone applying to teach. The database more than likely had a field composed of those last four digits. Unfortunately, the first aspect of “misconfiguration” was that the database also had the entire social security number. Typically, databases that provide data to the general public should not contain personally identifiable information (PII). Eliminating the full-length social security field should solve that problem.

Another possible element of the “misconfiguration” may have been in the script that delivers the result back to the inquirer. In this case, the script should have filtered out the first five digits of the social security number. This form of redaction is commonly done with your credit card number. The auto-fill fields only show the last three or four digits of the credit card number. This technique could have been intended by the web site designers in the Renaud case.

The Hacker

Alas, Josh Renaud is a “hacker.” Welcome to the club. It begs the question – “What is a hacker?”

To answer that question, I recommend you check out the latest cultural contributions to the term as described in Wiktionary. If you haven’t been in Wiktionary, it is the complement of Wikipedia. It focuses on developing a people’s dictionary. So it is rather interesting to look at the various definitions of “hacker,” ranging from gardening, to being clumsy in sports, to being curious and, in a narrow sense, to be deliberately intent on penetrating a secured network. Most people, including the governor and the commissioner of the Department of Education, consider a “hacker” in the latter sense of the word. In my circle, few who call themselves “hackers” are in that class. If you want to check out someone who has been there, done that, done their time, and have redeemed themselves, see Kevin Mitnick.

A large portion of IT personnel are probably “hackers” in a benign sense of the word. A hacker is essentially curious about how something works. If you do not have that curiosity, I find it mystifying why or even how you can remain in IT work. IT systems do not always work as expected. The answers are not found in a manual and may be impossible to dig up at chat sites or on Stack Exchange (see it for yourself). Renaud is no doubt in that class.

Hackers are generally a principled lot. As in Pirates of the Caribbean, they abide not by rules, but more like guidelines. The penultimate of hackerdom is the annual Black Hat convention. You can download tons of stuff at Black Hat either directly through their website or by following them on YouTube. Black Hat has a simple rule of ethics. You discover, you report to the owner, and then you publish. Not everyone follows those rules, unfortunately. But more do than you can imagine. It is the most critical aspect of IT security. There is not a system designer on the planet who can get it right the first time. The freelance hacker is often the first to discover a vulnerability, who has the skill level to understand the vulnerability, and the honesty to report it. With what I know so far, I would not put Renaud in the Black Hat class. But his approach to the problem fits the standard.

What Renaud has encountered, however, is the whistle-blower conundrum. The case of Edward Snowden is probably the top of the list here. If you report something, to whom do you report it and what are the consequences. If the State of Missouri decides to prosecute what is in essence a whistle-blower, what will be the consequences in the future when the next vulnerability is discovered – and NOT reported?

The Problem of Leadership

The governor of Missouri has joined the ranks of some prestigious personalities: Kathy Sibelius, Katherine Archuleta and Donna Seymour. Sibelius had the misfortune of having to be in the room when Obamacare was rolled out. Archuleta was at the head of Office of Personnel Management (OPM) when it experienced a massive data breach of personnel data of every federal employee in 2015, and Seymour was the CIO. I am sure all of these women are intelligent and capable. Their resumes are impressive by any standard. They were experienced in the business world as well as in politics and administration. But Lord have mercy when it comes to an IT event. As an instructor in Information Management, I had the privilege of having dozens of students in the classroom who were government administrators and military officers. Most of them struggled with the terminology of IT. It was not their thing. Yet, as successful administrators, they knew they had to develop the ability to understand the basics of IT and how to effectively build a team around them to tackle the challenges ahead. Otherwise, they may encounter the “deer in the headlights” moment when they are expected to explain an event and struggle to do so. These women went through those very uncomfortable circumstances when they were brought before Congress.

Governor Parsons has joined them. He is making what appears, with the information we have available to the public, a classic amateurish misstep when it comes to how IT works. I pity the guy in some sense because I have never expected politicians, business owners and administrators to understand everything about IT. But what should be expected is that they have the sense to put capable people around them who can explain what is happening. He evidently lacks that. I can only hope that the head of the Highway Patrol and the prosecutor of Cole County will have the sense to listen to their IT specialists.

The Failure of Change Management

Change Management is a term that applies to how changes are managed, whether it be a change to the design of a web site or to how your car is built. In my career, I participated in change management almost on a daily basis. The Renaud case demonstrates the pitfalls of IT programming. There are a lot of moving parts and it is very challenging to track them all. But just as a car is made of thousands of components and technologies, there is a process by which they are assembled and a process by which each step is checked. For assembled items, as they proceed down the line to the next phase, the item is checked by a human or electronics to verify that the previous step was completed correctly. Web sites are no different.

One thing that is abundantly clear is that the certification website skipped a quality control check. Personally Identifiable Infromation (PII) checks are standard procedure. This step was skipped or missed by the team that assembled the site. A PII review is usually a check-off item, meaning someone signs off on it. That person, in this case, has some explaining to do.

In regards to the “misconfigured” database, the designers put at risk PII by including it in a table of data that is streamed to the general public. This is not necessarily a failure of change management, but it is a question of design that should have caught the eye of someone on the database management team.

The Middle Ground

There are some alternatives to a criminal case of Josh Renaud. One is to borrow a page from the State of Alaska. They have set up a means by which users of their websites can use their MyAlaska accounts. These personal accounts are set up to handle the Permanent Fund filings (you know, the free money Alaskans get from the oil exported from the state). That account can also be used for several other services provided by the state. My guess is that Missouri uses such accounts somewhere, and they can expand that capability for the certification web page. If the state is so concerned about what people do on that site, requiring a login ID would help considerably with the security. A login ID would also enhance transparency for the teachers as e-mails could be sent to them whenever somebody inquires about their credentials. The IDs can also be used to qualify who is checking, restricting inquiries to users who are resident in the state or the United States.

The second alternative is to nest the page coding. If they are paranoid about anyone reading the page source code, then redesign the pages where its component parts are constructed through secured scripts. When I worked for the US Forest Service, I came across some pages that were about four lines of HTML code, each line referring to a secured script. Not much for anyone to see. The drawback is that the team of coders will have to be a tad bit more sophisticated to pull this off. It is also more resource intense.

What Happens if the State of Missouri Wins?

I seriously doubt this case will go anywhere. There is an outside chance that they may encounter a judge who is not only totally IT illiterate, but incapable of learning anything new. But beyond that, most judges will look at the evidence and write in the margins of the briefing “WTF” or “BSC”.

Source code will no longer be readable by users. Not sure how they are going to pull this off, but browsers would have to be redesigned to not show the code as easily. I doubt Google, Apple, Microsoft or Mozilla will oblige without one serious fight.
Missouri will have to petition international governing bodies to rewrite HTML standards. Again, that would require a national body to petition such a change.
Since the above stated changes will not be immediately forthcoming, the State of Missouri will have to rely on coercion to threaten anyone who reads HTML code with criminal prosecution. Since this is rather unique in the world, they will need to post a warning on each of their pages so that “hackers” will understand.
With the threat in place, it is highly unlikely the State of Missouri will receive a notification of any future security vulnerabilities. “Hackers” will most likely share the vulnerability amongst each other. The end result will be a security penetration that will go on for some time before anyone in the State’s IT group detect what is happening.
While scanning the literature, it is interesting to note that outsider reporting of vulnerabilities are often from major corporations providing a favor. Google, for example, reports numerous vulnerabilities to Microsoft regarding their products. Engineers from different firms will often share information to enhance security. The State of Missouri will have the arduous task of having prior agreements with anyone filing a report, and they will have to explain themselves when saying it is OK for Google to report a vulnerability, but an ordinary citizen cannot. Yet they may be reporting on the same thing.
Needless to say, the last thing that may fall to the ax is press freedom. Reporters will be advised to keep Edward Snowden posted in the event that anything they discover is traced back to them. With journalism going global, reporters will most likely direct the information to reporters residing outside the country. So the State of Missouri may be finding out by their vulnerabilities from Brazil or Russia, rather than an honest reporter from St. Louis.

In Conclusion

As you can see, the above listed ramifications would be quite dire and, thankfully in this country, highly unlikely. The job of reporting done by Josh Renaud was journalism at its best. It was thorough and it was responsibly reported. The State of Missouri and the Department of Education should quickly apologize and thank him for his efforts.

As regards the case itself, like almost all other IT security issues, public disclosure is sadly absent. So much of what we discuss is speculative. About the only thing the State of Missouri can effectively argue is that the “multi-step process” utilized techniques that are more than that employed by a journalist. So it brings to the question of whether Renard was acting alone and, if not, who was assisting. It also brings to the question of why he would employ diagnostic tools to analyze a state web site. Regardless, would discovery really matter if, in the end, he exposed private information was being funneled through a public network and reported it? It is difficult to state that what he did is criminal because he was presumably not employing “intrusive” techniques.

If there is any clear conclusion here, it is that the State of Missouri has a lot of explaining to do. If the State is to go to such efforts to pursue dubious prosecutorial objectives, then it really behooves the state authorities to explain how they failed in database design, web design and change management. They don’t need to name names – but they do need to present to the public what sort of concrete measures were taken to solve the problem.

Updates

February 12, 2022

Good news. According to the Springfield News-Leader, the Cole Country prosecutor decided to file charges.

February 16, 2022

A few tidbits about the “hack” was presented through MalwareBytes Labs.

First, the response of the Department of Education after Renard announced the exposure of social security numbers.

“Education Commissioner Margie Vandeven said in a letter to teachers that “an individual took the records of at least three educators, unencrypted the source code from the webpage, and viewed the social security number (SSN) of those specific educators.”

As noted in the article, how management and leadership understand IT is a crucial element in IT Security. It is obvious from the above response that the department leadership had very little understanding of the problem. People don’t “take” things that are publicly presented. Secondly, nothing in the HTML presentation is “encrypted”. MalwareBytes reports that the case actually got a name, the “F12 Case”, based on the use of the F12 function key to present web page source code.

Second element was even more surprising. The State of Missouri also pursued an expert who was consulted on the matter, Professor Shaji Khan of the University of Missouri – St. Louis. He was asked to review the exposure before the it was published.

Third item shows how ignorance of how IT functions can be extended into the political realm. Khan’s attorney provided a “hold and demand” letter to the State of Missouri detailing the technical aspects of the case and the state’s responsibility in its web development. But it also detailed damages to Khan’s reputation as well as that of Josh Renaud and the St. Louis Post-Dispatch when the case was extended to social media by a PAC, Uniting Missouri. They posted a video on YouTube regarding the case.

March 2, 2022

Kudos to Jhonti, who goes under the name of Seytonic. He has put together an excellent review of this case, providing visual examples of how the social security numbers were discovered.

As I have mentioned in many, if not all, of my cybersecurity-related articles, it is difficult to obtain information on the actual techniques used in a case. This video illustrates that the data was delivered in a basic encrypted form called Base64. It is easy to decrypt and is not really meant for sophisticated masking of data.

What this points to is the problem with database management. The encrypted string of data delivered to the web page contained the entire social security number when it was really not necessary. The solution is to 1) redesign the source table to not include personal identifiable information (PII), or 2)improve on the scripts that delivered the data to filter out PII. I would prefer the former simply because script writers can make mistakes in this regard, often focusing on the more technical aspects of coding and not on the data itself.

I could add a third element — improved change management that checks for PII exposure. It has been my experience that change managers struggle with technical aspects of IT and they may not have caught this type of exposure. But an experience like this is good training material because change managers can learn to ask the right questions to database managers and script writers.