Back in the mid-aughts, Adam G., a colleague on the IE team, used the email signature “IE Networking Team – Without us, you’d be browsing your hard drive.” And while I’m sure it was meant to be a bit tongue-in-cheek, it’s really true– without a working network stack, web browsers aren’t nearly as useful.
Background on Proxy Determination
One of the very first things a browser must do on startup is figure out how to send requests over the network. Typically, the host operating system already provides the transport (TCP/IP, UDP) and lower-level primitives, so the browser’s first task is to figure out whether or not web requests should be sent through a proxy. Until this question is resolved, the browser cannot send any network requests to load pages, sync profile information, update phishing blocklists, etc.
In some cases, proxy determination is simple— the browser is directly configured to ignore proxies, or to send all requests to a directly specified proxy.
However, for convenience and to simplify cases where a user might move a laptop between different networks with different proxy requirements, all major browsers support an algorithm called “Web Proxy Auto Discovery”, or WPAD. The WPAD process is meant to find and download a Proxy AutoConfiguration Script (PAC) for the current network.
Establish a HTTP(S) connection to discovered URL’s server and download the PAC script.
If the PAC script downloads successfully, parse and optionally compile it.
For each network request, call FindProxyForURL() in the PAC script and use the proxy settings returned from the function.
While conceptually simple, any of these steps might fail, and any failure might prevent the browser from using the network.
A Microsoft Edge feature team reached out to the networking team this week asking for help with an observed 3 second delay in the initialization of their feature. They observed that this delay magically disappeared if Fiddler happened to be running.
With symptoms like that, proxy determination is the obvious suspect, because Fiddler specifies the exact proxy configuration for browsers to use, meaning that they do not need to perform the WPAD process.
Note: Timestamps [e.g. t=52] are shown in milliseconds.
Because the browser took a full three seconds to decide whether or not to use a proxy, every feature that relies on the network will take at least three seconds to get the data it needs.
So, where’s the delay coming from? In this case, the delay comes from two places: a two second delay for PAC_FILE_DECIDER_WAIT and a one second delay for the DNS lookup of wpad.
The two second PAC_FILE_DECIDER_WAIT [Step #2] is a deliberate delay that is meant to delay PAC lookups after a network change event is observed, to accommodate situations where the browser is notified of a network change by the Operating System before the network is truly “ready” to perform the DHCP/DNS/Download steps of WPAD. In this browser-startup case, we haven’t yet figured out why the browser thinks a network change has occurred, but the repro is not consistent and it seems likely to be a bug.
The (failing) DNS lookup [Step #3] might’ve taken even longer to return, but it timed out after one second thanks to an enabled-by-default feature called WPADQuickCheckEnabled.
This three second delay on startup is bad, but it could be even worse. We got reports from one Microsoft employee that every browser startup took around 21 seconds to navigate anywhere. In looking at his network log, we found that the wpad DNS lookup [Step #5] succeeded, returning an IP address, but the returned IP was unreachable and took 21 seconds to timeout during TCP/IP connection establishment.
What makes these delays especially galling is that they were all encountered on a network that does not actually need a proxy!
Beyond the time delays, each of these steps might fail, and if a proxy is required on the current network, the user will be unable to browse until the problem is corrected.
For example, we recently saw that [Step #7] failed for some users because the Utility Process running the PAC script always crashed due to forbidden 3rd-party code injection. When the Utility Process crashes, Chromium attempts to bypass the proxy and send requests directly to the server, which was forbidden by the Enterprise customer’s network firewall.
WPAD is something of a security threat, because it means that another computer on your network might be able to become your proxy server without you realizing it. In theory, HTTPS traffic is protected against malicious proxy servers, but non-secure HTTP traffic hasn’t yet been eradicated from the web, and users might not notice if a malicious proxy performed an SSLStripping attack on a site that wasn’t HSTS preloaded, for example.
Edge Legacy and Internet Explorer have a surprising default behavior that treats sites for which a PAC script returns DIRECT (“bypass the proxy for this request“) as belonging to your browser’s Intranet Zone.
Interestingly, performance and functionality problems with WPAD might have been less common for the Edge Legacy and Internet Explorer browsers on Windows 10. That’s because both of these browsers rely upon the WinHTTP Web Proxy Auto-Discovery Service:
This is a system service that handles proxy determination tasks for clients using the WinHTTP/WinINET HTTP(S) network stacks. Because the service is long-running, performance penalties are amortized (e.g. a 3 second delay once per boot is much cheaper than a 3 second delay every time your browser starts), and the service can maintain caches across different processes.
Chromium does not, by default, directly use this service, but it can be directed to do so by starting it with the command line argument:
Prior to the enhancement of the WinHTTP WPAD Service, a feature called SmartWPAD was introduced in Internet Explorer 8’s version of WinINET. SmartWPAD caches in the registry a list of networks on which WPAD has not resulted in a PAC URL, saving clients the performance cost of performing the WPAD process each time they restarted for the common case where WPAD fails to discover a PAC file:
Cache entries would be maintained for a given network fingerprint for one month. Notably, the SmartWPAD cache was only updated by WinINET, meaning you’d only benefit if you launched a WinINET-based application (e.g. IE) at least once a month.
When a client (including IE, Chrome, Microsoft Edge, Office, etc) subsequently asks for the system proxy settings, SmartWPAD checks if it had previously cached that WPAD was not available on the current network. If so, the API “lies” and says that the user has WPAD disabled.
The SmartWPAD feature still works with browsers running on Windows 7 today.
Notably, it does not seem to function in Windows 10; the registry cache is empty. My Windows 10 Chromium browsers spend ~230ms on the WPAD process each time they are fully restarted.
If your computer is on a network that doesn’t need a proxy, you can ensure maximum performance by simply disabling WPAD in the OS settings.
On Windows, you can thus turn off WPAD by default by using the Internet Control Panel (inetcpl.cpl) Connections > LAN Settings dialog, or the newer Windows 10 Settings applet’s Automatic Proxy Setup section:
Simply untick the box and browsers that inherit their default settings from Windows (Chrome, Microsoft Edge, Edge Legacy, Internet Explorer, and Firefox) will stop trying to use WPAD.
WPAD is convenient, but somewhat expensive for performance and a bit risky for security/privacy. Every few years, there’s a discussion about disabling it by default (either for everyone, or for non-managed machines), but thus far none of those conversations has gone very far.
Ultimately, we end up with an ugly tradeoff– no one wants to land a change that results in users being limited to browsing their hard drives.
If you’re an end user, consider unticking the “Automatically Detect Settings” checkbox in your Internet settings. If you’re an enterprise administrator, consider deploying a policy to disable WPAD for your desktop fleet.
The past few weeks have seen a large number of new domain registrations beginning with the word “reopen” and ending with U.S. city or state names. The largest number of them were created just hours after President Trump sent a series of all-caps tweets urging citizens to “liberate” themselves from new gun control measures and state leaders who’ve enacted strict social distancing restrictions in the face of the COVID-19 pandemic. Here’s a closer look at who and what appear to be behind these domains.
A series of inciteful tweets sent by President Trump on April 17, the same day dozens of state-themed “reopen” domains were registered — mostly by conservative groups and gun rights advocates.
KrebsOnSecurity began this research after reading a fascinating Reddit thread over the weekend on several “reopen” sites that seemed to be engaged in astroturfing, which involves masking the sponsors of a message or organization to make it appear as though it originates from and is supported by grassroots participants.
The Reddit discussion focused on a handful of new domains — including reopenmn.com, reopenpa.com, and reopenva.com — that appeared to be tied to various gun rights groups in those states. Their registrations have roughly coincided with contemporaneous demonstrations in Minnesota, California and Tennessee where people showed up to protest quarantine restrictions over the past few days.
A “reopen California” protest over the weekend in Huntington Beach, Calif. Image: Reddit.
Suspecting that these were but a subset of a larger corpus of similar domains registered for every state in the union, KrebsOnSecurity ran a domain search report at DomainTools [an advertiser on this site], requesting any and all domains registered in the past month that begin with “reopen” and end in “.com.”
That lookup returned approximately 150 domains; in addition to those named after the individual 50 states, some of the domains refer to large American cities or counties, and others to more general concepts, such as “reopeningchurch.com” or “reopenamericanbusiness.com.”
Many of the domains are still dormant, leading to parked pages and registration records obscured behind privacy protection services. But a review of other details about these domains suggests a majority of them are tied to various gun rights groups, state Republican Party organizations, and conservative think tanks, religious and advocacy groups.
For example, reopenmn.com forwards to minnesotagunrights.org, but the site’s WHOIS registration records (obscured since the Reddit thread went viral) point to an individual living in Florida. That same Florida resident registered reopenpa.com, a site that forwards to the Pennsylvania Firearms Association, and urges the state’s residents to contact their governor about easing the COVID-19 restrictions.
Both the Minnesota and Pennsylvania gun advocacy sites include the same Google Analytics tracker in their source code: UA-60996284. A cursory Internet search on that code shows it also is present on reopentexasnow.com, reopenwi.com and reopeniowa.com.
More importantly, the same code shows up on a number of other anti-gun control sites registered by the Dorr Brothers, real-life brothers who have created nonprofits (in name only) across dozens of states that are so extreme in their stance they make the National Rifle Association look like a liberal group by comparison.
A number of other sites — such as reopennc.com — seem to exist merely to sell t-shirts, decals and yard signs with such slogans as “Know Your Rights,” “Live Free or Die,” and “Facts not Fear.” WHOIS records show the same Florida resident who registered this North Carolina site also registered one for New York — reopenny.com — just a few minutes later.
Merchandise available from reopennc.com.
Some of the concept reopen domains — including reopenoureconomy.com (registered Apr. 15) and reopensociety.com (Apr. 16) — trace back to FreedomWorks, a conservative group that the Associated Presssays has been holding weekly virtual town halls with members of Congress, “igniting an activist base of thousands of supporters across the nation to back up the effort.”
Many of the reopen sites that have redacted names and other information about their registrants nevertheless hold other clues, mainly based on precisely when they were registered. Each domain registration record includes a date and timestamp down to the second that the domain was registered. By grouping the timestamps for domains that have obfuscated registration details and comparing them to domains that do include ownership data, we can infer more information.
For example, more than 50 reopen domains were registered within an hour of each other on April 17 — between 3:25 p.m. ET and 4:43 ET. Most of these lack registration details, but a handful of them did (until the Reddit post went viral) include the registrant name Michael Murphy, the same name tied to the aforementioned Minnesota and Pennsylvania gun rights domains (reopenmn.com and reopenpa.com) that were registered within seconds of each other on April 8.
A large number of “reopen” domains were registered within the same one-hour period on April 17, and tie back to the same name used in the various reopen domains connected to gun rights groups. A link to the spreadsheet where this screen shot is drawn from is included below.
A Google spreadsheet documenting much of the domain information sourced in this story is available here.
No one responded to the email addresses and phone numbers tied to Mr. Murphy, who may or may not have been involved in this domain registration scheme. Those contact details suggest he runs a store in Florida that makes art out of reclaimed or discarded items, and that he operates a Web site design company in Florida.
However, various social media profiles tied to Mr. Murphy’s contact details suggest this persona may not present a complete picture. A Twitter account tied to Murphy’s email address promoted nothing but spammy paid surveys for years. And a Skype lookup on his phone number curiously returns a Russian profile under the name валентина сынах (translated as “Valentine Sons”).
As much as President Trump likes to refer to stories critical of him and his administration as “fake news,” this type of astroturfing is not only dangerous to public health, but it’s reminiscent of the playbook used by Russia to sow discord, create phony protest events, and spread disinformation across America in the lead-up to the 2016 election.
This entire astroturfing campaign also brings to mind a “local news” network called Local Government Information Services (LGIS), an organization founded in 2018 which operates a huge network of hundreds of sites that purport to be local news sites in various states. However, most of the content is generated by automated computer algorithms that consume data from reports released by U.S. executive branch federal agencies.
The relatively scarce actual bylined content on these LGIS sites is authored by freelancers who are in most cases nowhere near the localities they cover. Other content not drawn from government reports often repurpose press releases from conservative Web sites, including gunrightswatch.com, taxfoundation.org, and The Heritage Foundation. For more on LGIS, check out the 2018 coverage from The Chicago Tribune and the Columbia Journalism Review.
Palli Thordarson, chemistry professor at the University of New South Wales, writing for The Guardian:
Viruses can be active outside the body for hours, even days.
Disinfectants, liquids, wipes, gels and creams containing alcohol
are all useful at getting rid of them — but they are not quite as
good as normal soap.
When I [shared the information above using Twitter][t], it went
viral. I think I have worked out why. Health authorities have been
giving us two messages: once you have the virus there are no drugs
that can kill it or help you get rid of it. But also, wash your
hands to stop the virus spreading. This seems odd. You can’t, even
for a million dollars, get a drug for the coronavirus — but your
grandmother’s bar of soap kills the virus.
So why does soap work so well on the Sars-CoV-2, the coronavirus
and indeed most viruses? The short story: because the virus is a
self-assembled nanoparticle in which the weakest link is the lipid
(fatty) bilayer. Soap dissolves the fat membrane and the virus
falls apart like a house of cards and dies — or rather, we should
say it becomes inactive as viruses aren’t really alive.
I was not aware until this week that good old-fashioned soap is significantly more effective than alcohol-based disinfectants.
That list of supported platforms is missing a big name: Nintendo Switch. Tuesday's delay includes an additional, indefinite delay of the sequel's port to Nintendo's weaker console, thus breaking the developer's original promise that Switch buyers would get to rip and tear into Doom Eternal the same day as everyone else. "We will announce [the Switch port's] date in the future," the company's statement vaguely reads.
Publisher Bethesda took the opportunity to delay another related game out of November 2019, as well: Doom 64. This first-ever port of the 1997 shooter onto non-N64 platforms is still coming to PC and modern consoles, Bethesda says, but it too will launch on March 20, 2020. Now, at least, that port will become a free pre-order bonus for buyers of Doom Eternal. But we're not sure why Bethesda and id Software couldn't get Doom 64 ready by this holiday season to tide series fans over during the bigger game's delay. (In the meantime, if you own a legitimate copy of the N64 original, we suggest ripping its files and launching them on PC via the incredible Doom 64 EX mod.)