A Google Chrome experiment has gone horribly wrong this week and ended up crashing browsers on thousands, if not more, enterprise networks for nearly two days.
The issue first appeared on Wednesday, November 13. It didn't impact all Chrome users, but only Chrome browsers running on Windows Server "terminal server" setups -- a very common setup in enterprise networks
Complaints flooded Google
According to hundreds of reports, users said that Chrome tabs were going blank, all of a sudden, in what's called a "White Screen of Death" (WSOD) error.
The issue was no joke. System administrators at many companies reported that hundreds and thousands of employees couldn't use Chrome to access the internet, as the active browser tab kept going blank while working.
In tightly controlled enterprise environments, many employees didn't have the option to change browsers and were left unable to do their jobs. Similarly, system administrators couldn't just replace Chrome with another browser right away.
"This has had a huge impact for all our Call Center agents and not being able to chat with our members," someone with a Costco email address said in a bug report. "We spent the last day and a half trying to figure this out."
"Our organization with multiple large retail brands had 1000 call center agents and many IT people affected for 2 days. This had a very large financial impact," said another user.
"Like many others, this has had significant impact on our organization with our entire Operations (over 500 employees) working in a RDS environment with Google Chrome as the primary browser," said another system administrator.
"4000 impacted in my environment. Working on trying to fix it for 12 hours," said another.
"Medium sized call center for a local medical office lost a day and a half of work for 40-60 employees," added another.
"Same issue experienced, hundreds of users impacted - hours spent attempting to isolate the cause," said another user.
Hundreds of complaints poured in via Google's support forum, Chrome bug tracker, and Reddit [1, 2]. One impacted sysadmin told ZDNet that they initially mistook the Chrome blank tabs as a sign of malware and reacted accordingly, starting network-wide security audits.
Google ships a fix
However, with time, the root cause of the bug was eventually found, and traced back to a feature called "WebContents Occlusion."
According to Google Chrome design document, this is an experimental feature that suspends Chrome tabs when users move other app windows on top of Chrome, treating the active Chrome tab as a background tab.
Image: Google
The feature, meant to improve Chrome's resource usage when not in active use, had been under testing in Chrome Canary and Chrome Beta releases all year.
However, this week, Google decided to test it in the main Stable release, so it could get more feedback on how it behaved.
That it behaved badly is an understatement.
"The experiment/flag has been on in beta for ~5 months," said David Bienvenu, a Google Chrome engineer. "It was turned on for stable (e.g., M77, M78) via an experiment that was pushed to released Chrome Tuesday morning."
"Prior to that, it had been on for about 1% of M77 and M78 users for a month with no reports of issues, unfortunately," he added.
However, when rolled out to a broader audience -- such as Windows users on terminal server setups -- an unexpected bug occurred that instead of suspending Chrome tabs when users switched to another app, it unloaded the tab entirely, leaving a blank page behind.
Users could refresh the Chrome tab to access their sites again, but in some cases, this also meant they lost previous work.
The Chrome team said they pushed a new Chrome configuration file to all Chrome users and disabled the experiment.
Chrome engineers operate a system called Finch that lets them push updated Chrome settings to active installs, such as enabling or disabling experimental flags.
If the fix has not reached all impacted users, and they still have problems, they can disable the following two experimental flags by hand:
An alternative method to fixing this is to start Google Chrome with the following command-line argument: --disable-backgrounding-occluded-windows
Fix prompts more criticism
However, fixing the problem actually made system administrators even angrier. Many didn't know that Chrome engineers could run experiments on their tightly-controlled Chrome installations, let alone that Google engineers could just ship changes to everyone's browsers without any prior approval.
"Do you see the impact you created for thousands of us without any warning or explanation? We are not your test subjects," said an angry sysadmin. "We are running professional services for multi million dollar programs. Do you understand how many hours of resources were wasted by your 'experiment'?"
"How many tens of thousands of dollars has this oops cost everyone? This is starting to look like a pretty massive mistake on Googles part," added another disgruntled sysadmin.
"We take great care in rolling our changes out in a very controlled manner to avoid this type of scenario and we spent the better part of yesterday trying to determine if an internal change had occurred in our environment without our knowledge. We did not realize this type of event could occur on Chrome unbeknownst to us. We are already discussing alternative options, none of them are great, but this is untenable," said another, hinting at a browser change across their organization.
Although it lasted just two days, this entire incident is panning out to be one of the Chrome team's biggest bungles. Many impacted users demanded an official apology from Google, and by the looks of the financial impact it may have caused some companies, they are entitled to it.
David Beckham. Paris Hilton. Bono. Everyone who was anyone in the early 2000s owned a Motorola Razr. With its sleek edges and metallic keyboard, it stood out from a crop of generic flip phones. And it wasn't just for the elite. In its four-year run, the chic, ultra-thin handset sold 130 million units on its way to becoming one of the most beloved and iconic pieces of technology ever.
Then Apple's iPhone came along and absolutely destroyed it, along with the memory of so-called dumb phones.
Now more than a decade later, Motorola, which invented the cell phone but toils in mobile obscurity, is staging a comeback with a reimagined Razr. It combines the retro clamshell design with Google's Android software and an innovative foldable touchscreen display that puts a new twist on the classic flip phone. CNET reviewer Jessica Dolcourt, after an early and extended look at the Razr, describes it as "Streamlined. Utterly pocketable. Nostalgic, with a sharp futuristic edge."
At $1,500, the new Razr is too expensive to go after even premium phones, including the iPhone 11 and Samsung's Galaxy S10. But the Bunjac and company are hoping that its ability to stand out from the pack -- once again -- will bring buzz back to Motorola and get consumers excited about foldables.
We could use the jolt. Foldable phones arrived with massive hype and high expectations earlier this year. They were supposed to reinvigorate the wireless world and represented a potential sea change in how we use our mobile devices. The new category was also going to serve as a bridge to the kind of sci-fi tech we see in films such as Minority Report or shows like Westworld.
The new Razr is a reinterpretation of the classic ultra-thin flip phone.
Angela Lang/CNET
Unlike those two devices, which are positioned as the future of their respective franchises, Motorola is reaching back to one of its most well-known icons as a way into the market for super-premium smartphones. It's hoping that the nostalgic chords it tugs at, combined with a seamless 6.2-inch display (when unfolded) and a compact design (when folded), can generate awareness not just for the Razr, but its larger stable of phones.
Motorola, which has about 3% of the handset market, faces an uphill climb. The company now specializes in budget phones, so price remains a question. It's unclear whether that Razr name, so beloved by an older generation of consumers who grew up on flip phones, will even register with a majority of younger users, who view the first iPhone as the iconic phone.
Then there's the question about durability.
What's certain is Motorola and Lenovo have large ambitions with the Razr. They've placed a unique bet on foldable screens, making the device smaller as opposed to Samsung's and Huawei's devices, which unfold into larger tablets.
"If the goal is to create a splash with a foldable phone, that could be very effective," says Avi Greengart, an analyst at consumer research firm Techsponential. "Anything Motorola can do to break through the clutter and remind people that they still make desirable phones will only help."
Back to the future
"We didn't know back then we were creating these iconic devices," says Mike Jahnke, global industrial designer at Motorola.
Mike Jahnke, global industrial designer at Motorola, worked on both the original and new Razr.
James Martin/CNET
I'm chatting with the 21-year Motorola veteran at the phone maker's headquarters in the historic Merchandise Mart building in Chicago. Jahnke is decked out in a matching purple patterned bow tie and sweater, which pop in front of the gaggle of different-colored Razr phones behind him.
As one of the designers of the original Razr, Jahnke shares stories about celebrities who dropped in requesting special variants of the flip phone. The device had taken off as a status symbol after Motorola took the savvy step of slipping a special black edition of the Razr into the swag bags of the 2005 Academy Awards. Shortly after, every power player needed to have one -- customized to their tastes.
After its initial run as a $600 exclusive at wireless provider Cingular, Motorola added new carriers, new variants and dropped the price.
Motorola pulled off the then-impressive task of making a flip phone cool and accessible.
By 2006, the company owned more than a fifth of the global handset market, according to Neil Shah, a Counterpoint Research analyst. The Razr spawned a number of imitation thin phones from rivals such as Samsung and Sanyo's Katana. Motorola tried to follow up the Razr's success with phones like the Rizr or Slvr, which featured a similar aesthetic. Soccer legend David Beckham served as the face of the Razr 2.
"Everybody else had to struggle to catch up," said Roger Entner, a consultant for Recon Analytics. "The copycat devices turned the Razr into a more iconic device."
It was around the time that Google took the reins that Motorola started playing around with plastic as a material for the display. Most phone makers used glass because it offered more heft and a premium feel. But plastic could bend and flex, and when treated with a special coating, proved to be surprisingly durable.
So despite a rich history of landmark devices, from the first cell phone, the DynaTac, to the first flip phone in the StarTac, it was the Droid Turbo 2 that may have provided the key inspiration for the new Razr.
I'll forgive you if it doesn't immediately spring to mind.
Motorola and Lenovo started thinking about foldable around 2015.
James Martin/CNET
Around the same time, engineers and designers were looking at foldable devices and started collaborating with their counterparts at Lenovo, whose Japanese unit lent its expertise in hinge designs used in its YOGA laptops. The teams showed off the early fruits of that endeavor at Lenovo's Tech World in 2016, which included a phone that wrapped around your wrist. The working prototype still sits in Motorola's design labs.
"The new Razr is a design and innovation icon reinvented," Lenovo CEO Yang Yuanqing says in a statement. "It is also a powerful reflection of the engineering and technology prowess that exists across our company."
Internally, Motorola went through more than 20 different prototypes, and by the end of 2016, settled on a phone that folded down into a smaller device. Motorola executives saw the similarities between the final prototype's clamshell design and the original Razr.
All of a sudden, the comeback was on.
Motorola played with many prototypes before settling on the final design.
James Martin/CNET
"We didn't directly set out to do the Razr, but it made sense," Ruben Castano, vice president of design for Motorola, says as he stood over a large pile of different prototypes of the new Razrs.
The team even took a page from the original Razr's design scheme of shoving components into its thicker chin at the base. This time around, Motorola had a slew of additional hardware to add, including antennas for 22 bands of cellular airwaves for 4G LTE (there's no 5G connection), a fingerprint sensor, Wi-Fi radio, GPS and more.
Over the next two years, Motorola designers and engineers quietly worked on the project. In 2017, Motorola brought an early prototype to Verizon, which snapped it up as an exclusive.
"It's one of those devices you put down at the dinner table and it's an eye-catcher," says Brian Higgins, vice president of device and consumer product for Verizon Wireless.
The timing was auspicious. Excitement over foldable phones hit a fever pitch after Samsung led off its Unpacked presentation with a tease for the Galaxy Fold, stealing the spotlight from its flagship Galaxy S10 smartphone. A few days later, Huawei floored attendees of the Mobile World Congress trade show in Barcelona with the Mate X. The market for these devices – despite their prices – was shaping up.
If Motorola's executives had freaked out over the Samsung incident, they didn't show it when I asked one after another about their reactions.
"I don't see anything surprising," Buniac says. "That's what new technology is all about, solving issues."
Castano said the news assured him that his team was on the right track, since one of his early prototypes looked similar to the Fold.
But Verizon's Higgins acknowledged there was an impact. "When the reports came out, I immediately went back to Motorola," he says, noting he had the company re-run all of its specification tests.
Samsung ultimately came back with a reinforced Galaxy Fold, but the reputation for folding phones had taken a hit.
What had Motorola feeling confident was the unique hinge design it created for the new Razr. The ultimate goal was to create a mechanism that would allow the screen to close in on itself. But here's the dilemma: If you fold a sheet of plastic or paper completely flat, you inevitably get a crease.
That's why the Galaxy Fold, which folds inwards, has a large gap in the middle. It's also why Huawei chose to position its folding screen on the outside. By increasing the folding radius, you decrease the stress on the display and the likelihood of a crease.
It wasn't an easy process. Early prototypes had the phone with giant hinges jutting out from each side, giving it Dumbo-like ears that would've ruined the aesthetics.
You can see the exposed gears of the hinge on each side of the new Razr.
James Martin/CNET
Finally, the designers were able to accomplish a complete fold with what they call the zero-gap hinge. Like the earlier prototype, the hinges remain at the side, but not as far out. In fact, you can see some of the gears exposed, offering a glimpse into the mechanics of the construction, similar to a luxury watch. It also added space for the display to bend inward. If you could look into a cross section of the phone when its folded, you would see the screen bend in an intricate teardrop shape rather than a flat fold.
Tom Schirtzinger, director and principle engineer of innovation and architecture for Motorola, walked us through the different components of the Razr, noting that the display panel, which featured five layers, including one made of stainless steel, was only 360 microns thick.
Schirtzinger, who went from college straight to Motorola's R&D team 23 years ago, also said that a similar hard coat used in the Shattershield display was added to the Razr display so it could withstand the scratches that plastic displays are prone to get. But the team dialed back the coating to allow for more flexibility.
Motorola also added steel plates behind the display on each side of the bend point, reinforcing the display and straightening out any creases, a step Samsung took with its reinforced Galaxy Fold.
"We looked at the ways this could fail and would account for it," Schirtzinger says.
Bigger isn't always better
Unlike Samsung or Huawei, which showed their products early, Motorola has kept the Razr close to the vest.
Motorola had initially set a target to launch the Razr in the summer. When asked about the delay, Buniac cracks, "I'm Brazilian, so my summers come in January," before acknowledging that the company missed its deadline.
"Of course, we want to be fast to market, but we are not in a hurry or desperate to launch so we believe now is the right time," Buniac says.
Beyond timing, the Razr stands out from the Mate X and Galaxy Fold because its design allows the phone to go from normal size to small. Laura Joss, global director of design research, says that consumer surveys showed the contradicting trend of people wanting larger displays, but not the massive devices required to house them.
Motorola is positioning the Razr as the ideal compromise.
The clamshell design meant Motorola could incorporate a second, 2.7-inch outer "Quick View" screen when folded, just like the original. It's designed to handle notifications, control your music and work as a selfie display, since the main 16-megapixel camera is housed right underneath.
The Quick View screen offers easy access to notifications.
James Martin/CNET
Jeff Snow, general manager of innovation products at Motorola, says he often doesn't even flip open his Razr, which he's been using since July. Instead, he opts to triage his messages and other notifications from the tiny secondary screen.
Within a year, flip phones could become the new black.
Tapping into nostalgia
The original Razr design, with those metallic keys and that 10 millimeter-thick chassis, still evokes an emotional reaction when you hold one. It's a callback to a simpler time when we actually used our phones for phone calls.
It's also a reminder that Motorola was once a titan of innovation, pouring the kind of massive dollars into R&D that only the likes of Apple and Google are able to do now.
Motorola is hoping the new Razr will stir some of those memories now. It even got some of the little details down, like the satisfying sound the phone makes when you snap it shut -- hanging up on someone hasn't felt this good in a long time.
The Razr isn't a literal interpretation of the classic phone. But it shares many of the same traits.
James Martin/CNET
While the Razr brand does strike a nostalgic chord among older consumers, fond memories may not be enough to justify the $1,500 price tag.
"Nostalgia can only get you so far," says Ramon Llamas, an analyst at research firm IDC.
Note: While the NES Classic and SNES Classic were hits for Nintendo, Sony's own retro play with the PlayStation Classic was a flop. Capitalizing on our rose-tinted memories is no guarantee of success.
The Razr brand may not even register with younger users, with smartphones all but wiping dumb phones from our memories. Carolina Milanesi, an analyst at Creative Strategies, said she would recommend that Motorola emphasize the new design and the foldable trend rather than dwell on a past fad.
"The Razr was all about design," she says. "People used the Razr because it was cool, not because it was great."
Motorola says it expects the Razr to last the average lifespan of a smartphone but wouldn't specify an exact length of time. To help assuage customers, the company will offer a 24-hour turnaround time on replacements if the unit or display fails, although it didn't say how long the warranty lasts.
The business has been on an upswing, having posted four quarters of profitability after years of bleeding. Still, it will end the year with about 3% of the market -- largely driven by budget phones -- underscoring that it is a far cry from heavy hitters Samsung or Huawei. While Buniac is starting to turn his focus towards growing the business and introducing more premium devices, he has realistic expectations for the Razr.
"I don't see it as a silver bullet," Buniac says. "We see it as a journey that's just started."
This time around, the new Razr likely won't have celebrities lining up to nab an exclusive edition. When asked if Beckham would return with the new foldable version, Jhanke retorts, "We'd be happy to sell him one."
This story published at 8:11 p.m. PT.
Motorola hopes the Razr name will resonate with some consumers. But some analysts believe the company should focus on the new design.
SpaceX has successfully completed a round of Crew Dragon tests, which previously ended in an explosion. The space company attempted a similar set of static fire engine tests on April 20th, but it admitted a few days later that the Crew Dragon capsule it used was destroyed. It blamed the explosion on a component that leaked liquid oxidizer into high-pressure helium tubes. The more recent attempt went off without a hitch.
The company conducted the tests near Landing Zone 1 on Cape Canaveral Air Force Station in Florida, starting with two one-second burns for two of the Crew Dragon's 16 Draco thrusters. Those thrusters will be used for maneuvering and attitude control, as well as for re-orientation of the spacecraft. The Crew Dragon team then fired up eight SuperDraco engines, which will be in charge of accelerating the spacecraft away from its launch vehicle, for nine seconds.
Full duration static fire test of Crew Dragon's launch escape system complete – SpaceX and NASA teams are now reviewing test data and working toward an in-flight demonstration of Crew Dragon's launch escape capabilities pic.twitter.com/CMHvMRBQcW
After that, the Draco thrusters and SuperDraco engines fired in the same sequence they would if the spacecraft needed to reorient itself in-flight to be able to deploy its parachute and close its flaps prior to atmosphere reentry. These tests will help SpaceX and NASA validate Crew Dragon's launch escape system before a planned in-flight demonstration. Before that demonstration happens, though, both NASA and SpaceX need to review the data from these tests, perform hardware inspections and establish a date for the in-flight abort test.
All products recommended by Engadget are selected by our editorial team, independent of our parent company. Some of our stories include affiliate links. If you buy something through one of these links, we may earn an affiliate commission.
SpaceX has successfully completed a round of Crew Dragon tests, which previously ended in an explosion. The space company attempted a similar set of static fire engine tests on April 20th, but it admitted a few days later that the Crew Dragon capsule it used was destroyed. It blamed the explosion on a component that leaked liquid oxidizer into high-pressure helium tubes. The more recent attempt went off without a hitch.
The company conducted the tests near Landing Zone 1 on Cape Canaveral Air Force Station in Florida, starting with two one-second burns for two of the Crew Dragon's 16 Draco thrusters. Those thrusters will be used for maneuvering and attitude control, as well as for re-orientation of the spacecraft. The Crew Dragon team then fired up eight SuperDraco engines, which will be in charge of accelerating the spacecraft away from its launch vehicle, for nine seconds.
Full duration static fire test of Crew Dragon's launch escape system complete – SpaceX and NASA teams are now reviewing test data and working toward an in-flight demonstration of Crew Dragon's launch escape capabilities pic.twitter.com/CMHvMRBQcW
After that, the Draco thrusters and SuperDraco engines fired in the same sequence they would if the spacecraft needed to reorient itself in-flight to be able to deploy its parachute and close its flaps prior to atmosphere reentry. These tests will help SpaceX and NASA validate Crew Dragon's launch escape system before a planned in-flight demonstration. Before that demonstration happens, though, both NASA and SpaceX need to review the data from these tests, perform hardware inspections and establish a date for the in-flight abort test.
All products recommended by Engadget are selected by our editorial team, independent of our parent company. Some of our stories include affiliate links. If you buy something through one of these links, we may earn an affiliate commission.
