On Monday I read this blurb from the AP that Arizona Secretary of State Michele Reagan’s office says her technology team has launched an enhanced website to show voting returns from all 15 counties Tuesday night in real-time. New Arizona election reporting system to debut Tuesday.
My first thought was, “Oh no! Something else Michele Reagan can screw up this year.”
So how did it go? Just as you would expect from the queen of election screw-ups. ‘Error’: Arizona’s new elections site struggles in debut:
The Secretary of State’s Office, which oversees statewide elections, was supposed to be issuing the early results of races across the state close to 8 p.m. Tuesday.
Instead, the website produced this message: “Error.”
The website continued to intermittently produce results throughout the night, occasionally producing messages saying, “This site can’t be reached” and “Service unavailable.”
A spokesman for Secretary of State Michele Reagan, when asked about the error messages around 8 p.m., told The Arizona Republic the office was “working on” the issues.
Hours later, Matt Roberts told The Republic he couldn’t say with certainty the root of the problems. He said it was unclear whether website traffic contributed to the problem, but issues uploading files from some county recorders delayed results.
You probably should start with your office’s “technology team” that built your web site.
The secretary of state’s problems come after the office built a system in-house following troubles with a prior vendor. By moving the system in-house, Roberts estimated the office saved $200,000 when compared with the office’s old contract for the services, for a total cost of $751,000.
Roberts said the officials had no regrets over changing the system: “Anytime you launch a new system, there’s going to be hiccups and we certainly experienced that right out the gate.”
Even before election night, Reagan said she was anxious about the website, telling the newspaper: “On election night, I’ll either be opening a bottle of champagne or a bottle of Jack Daniels.”
Drinking on the job? Really? Is this another problem?
UPDATE: The Secretary of State’s office claims to have found the problem in its new web site. Why you waited for election results Tuesday night:
The slowdown in reporting election results Tuesday night was due to a faulty automated email the office generated, blocking feeds into the Arizona secretary of state’s online system, the office said Wednesday.
The system repeatedly displayed error messages just as early returns were released, and as candidates, the media and the public were visiting the site. The results displayed sporadically until late evening, although spokesman Matt Roberts said the problem was resolved in the 9 o’clock hour.
Roberts said the office sends an XML file with election results to media outlets. It automatically resends the email as new results come in. But because the Secretary of State’s Office’s new system didn’t test “as rigorously as it could have” for that interaction, it created an unexpected jam, Secretary of State Michele Reagan said in a blog post on the Secretary of State’s Office’s website.
Staffers who developed the system, which was built in-house, figured out a workaround that allowed election results to feed into the online system. It was working smoothly by late evening.
Meanwhile, the staff also has fixed the problem with the automated emails, so the blockage should not recur, Roberts said.
Well Mr. Huppenthal you must have had the magic connection…because all I was ever able to get all evening was a blank white page, and eventually, a timeout message.
That said, (and this kind of thing is also part of my day job) I more than half expected this. It’s very difficult, unless you’ve done it before more than once, to build out a website that will get hammered like this on it’s debut.
It’s hard to simulate hundreds of thousands of hits a minute, and if this is not your usual area of expertise, you find out very quickly how unforgiving the Internet can be.
The true test will be how it holds up in November, now that they’ve had their first “public beta”, so to speak. There’s a reason
that experts are usually hired for this sort of thing….
BruceJ, not to get into an architectural debate, but it’s actually very simple to test traffic to what is basically a static webpage.
And whatever caching solution they’re using (I assume they have one, otherwise fire them all) takes all the traffic while the web server and database sleep in the background, anyway. There are no transactions.
None of that matters though, because the problem as described by Reagan on her blog, linked above, is a locked file.
How do you have a test and not find a locked file?
The answer is, they didn’t test properly.
Poor management.
I looked at the site numerous times during the night and never had a problem.
Also, every Fortune 500 company has written off a 50 million dollar project or bigger that completely wiped out.
California had a billion dollar judicial software development project that imploded.
Fifteen counties, hundreds of voting sites, scores of elected officials – problems were guaranteed.
Sorry, no.
When a 50 million dollar project fails, it’s because they ran out of money, or the executives no longer want to fund/or need the project, they picked the wrong vendor(s), or they just bit off more than they can chew.
These reasons always fall under mismanagement. Always.
This was a simple project, collecting numbers and showing them on a webpage. It is not rocket science.
This is what I do for a living, Falcon9, it’s not my opinion that you are wrong, you are wrong.
Oh, and one more thing, when a large scale or high visibility IT project fails, someone loses their job.
Reagan failed, multiple times, she needs to go.
Okay, this is my wheelhouse. This is exactly what I do for a day job.
The statement from Reagan that “Anytime you launch a new system, there’s going to be hiccups and we certainly experienced that right out the gate” is false.
In fact, I’ve worked at places where even thinking about saying that out loud would get you fired. Think system failure = stock price drops, shareholders sue kind if stuff.
This is a small potatoes system and it’s simple to load test. This is 2016, we are not blazing news trails or using bleeding edge technology much these days.
The only reason a small potatoes system like this failed is a failure to properly test, a lack of proper QA/UAT, and that’s a management call.
And she’s the manager.
The same people that condemned the website rollout issues with the ACA are now making an exception.