Research methods

In July 2016, 18F and FEC staff participated in a workshop dedicated to understanding the principal goals of the e-filing study and to surface knowledge about the shape of the study. The three main goals of the workshop were to: outline the hopes and fears for the study, get an introductory understanding of the processes that shape e-filing, and, determine the key groups to talk to. The full workshop read-out is available here, but some takeaways follow. Some of the major hopes of the study were to increase the overall accuracy of the filings and to understand how we might help address the burden of dealing with paper forms from the senate.

Participants expressed anxiety that the changes wouldn’t be allowed, or that that it would introduce security threats or take too long. Participants also outlined some of the most important users to talk to, including FEC’s RAD analysts, vendors, and of course the people who file reports to the FEC (henceforth referred to as filers). These hopes, fears, processes and research subjects gave us a strong introduction to the project to build our research plan upon.

We conducted our research in three phases. In the first phase we conducted background research, interviewed and observed stakeholders to understand the current e-filing process and areas for improvement. In phase 2 we developed design concepts that we believed had the potential to make the process better, and then we had users evaluate these concepts via usability testing. In phase 3 we conducted a survey of all e-filing users to learn more about how they would feel about specific changes the FEC might want to make.

Phase 1: Contextual interviews

To understand how to improve the FEC’s electronic filing system for the different people who interact with it, we first had to talk with those people.

The following research questions guided our work:

• How do filers currently use e-filing software? What aspects of their process are particularly cumbersome or error prone? What barriers drive them crazy? What short-cuts, workarounds, and help do they employ? What’s working well?
• E-filing analysts (internal to the FEC): how does the form and format of e-filing negatively and positively affect the Electronic Filing Office and RAD analysts’ workflow?
• How do different kinds of vendors interact with their filers and the FEC? How might software vendors be impacted by changes to the FEC’s filing process?
• We also interviewed FEC technologists and contractors on the current system to ask them; What is going well? What are the pain points?

We worked with our agency partners to generate an email list of potential participants from the FEC’s e-filing database. We identified additional participants via an online screener on the beta FEC website.  In phase 1, we conducted semi-structured interviews with filers, commercial software vendors, analysts in the FEC’s Reports Analysis Division, and other FEC stakeholders. The focus of the interviews was on participants current processes including pain points and information gaps. Interviews with filers and RAD analysts included a portion where we could observe participants on their own computer using their own systems to complete some actual filing work. In most cases, these interviews were conducted remotely using online screen sharing and audio conferencing software, which were screen and audio recorded.  In some cases we conducted the interviews in-person and documented these interviews in field notes.

Following each session, we came together to talk through what we were seeing and synthesize our knowledge in the form of additional notes, whiteboarding sessions, and/or sketches. Our research team systematically reviewed field notes, videos, and other work artifacts gleaned during these sessions and interviews to identify themes. We then independently “tagged” quotes from users that are evidence to these themes. We then articulated our riskiest assumptions and developed hypotheses that we could test through prototype testing in phase 2.

Phase 2: Prototyping and testing

In phase 2, we sought to validate the hypotheses developed in phase 1 through the creation and testing of prototypes.  A prototype is basically a draft site built quickly and roughly with throw-away code that can be used to observe user interaction with examples of potential sites. It is built with a priority on quick and simple development for rapid learning and iteration, so it is not built in a way that it can serve as the foundation for future iterations of a production site. The results of prototype testing informed our recommendations and study findings.

A list screencasts demonstrating some of the features we tested in our work-in-progress prototypes are below:

• Manual data entry flow
  • Detail: in-line field level validation on the date field
  • Detail: in-context glossary
• Importing data from a spreadsheet flow
• Merging duplicate name records flow

Additionally, the code used for prototyping is available on GitHub.

Phase 3: Survey

Phase 3 comprised the distribution and analysis of a survey to help us test one of our hypotheses that proved difficult to explore via prototyping. Specifically, we needed to understand the filing community’s thoughts around a web-based version of FECFile.

Methods

We conducted a survey to find out how the filing community felt about the possibility of a web-based version of FECFile. Participants were invited to the survey via an email sent from the FEC’s Electronic Filing Office (EFO) that linked to a web-based survey form.

Our partners at the FEC helped us compile the survey distribution list by providing us with the committee IDs for all committees who filed electronically in Q3 2016 and all of the email addresses associated with these committees. In many cases, the same email address was listed several times for the same committee or for several different committees. After we removed duplicate email addresses, our distribution list comprised 8,906 unique email addresses. We received 533 responses.

Part of our intention was to make sure participants were comfortable sharing their opinions, even if they might be perceived as unpopular by others in the filing community, so we kept the survey anonymous. We were able to distribute the survey to the entire population of committees who had recently filed electronically, and we minimized sampling error by reaching more than our target number of 361 responses to be able to make population level inferences with a 95% Confidence Interval (CI) (margin of error, 5%).  We avoided coverage error by distributing the survey to all committees who filed electronically in the most recent quarter.

Analysis

One researcher combed through qualitative responses tagging the data  for prevalent themes. Some of the themes were mentioned by a large number of people, and some of them were mentioned less frequently. Additional analysis involved looking for themes within subsections of the population. The qualitative findings are organized by the questions we asked.

Limitations

No survey is without limitations. In a few cases, we asked participants to self report estimates, so these answers could be subject to recall bias. The survey was voluntary, so there could be some some self-selection bias. Specifically, the email invitation mentioned that the survey was exploring possible changes to FECFile, so we may have received a disproportionate number of responses from opinionated FECFile users. Similarly, there could be some non-response bias because filers who do not use FECfile might have been less likely to participate.

Overall study limitations

One of our recommendations throughout this report is that the FEC adopt an agile approach to technology transformation. Working in an agile way means planning, conducting, and evaluating work in short sprints and revising plans as needed. Adopting an agile mindset means embracing the fact that plans must always change and that laying out lengthy detailed plans before the work begins can mean of lot of wasted effort. In keeping with the spirit of an agile approach, our aim here is to provide the FEC with what it needs to begin the journey of modernizing the current e-filing system rather than provide an exhaustive list of all that this journey will entail. We see this as a strength rather than a limitation, but recognize that this is not typical for many government reports that may be more familiar to some readers.

In keeping with this agile mindset, we worked with our FEC partners to prioritize who to talk to within the study timeframe. While we did not interview members of some stakeholder groups in this initial discovery project, we strove to make it clear throughout the report the user groups that must be involved in the iterative design, development, and evaluation of any changes that the FEC decides to make.

What is your role in filing reports to the FEC?

 On behalf of whom do you typically file?

How many transactions do your reports typically include?

*Numbers may not total 100% due to rounding.

**Unanswered (n = 2); Other (n = 99): Prevalent answers were assistant to a treasurer, PAC administrator, bookkeeper/accountant, and campaign manager.

†Unanswered (n = 4); Other (n = 14): Prevalent answers were PAC, assistant, analyst, person responsible for ensuring reports are filed but not the person who actually completes them.