Coinbase Exchange Inaccessible Due To 5x Traffic Spike During Bitcoin Surge
According to Coinbase, the exchange’s autoscaling couldn’t keep up with a 5x traffic spike experienced during a Bitcoin price surge to $10,000. Coinbase Exchange Inaccessible Due To 5x Traffic Spike During Bitcoin Surge
The largest cryptocurrency exchange in the United States said its autoscaling was unable to keep up with a huge traffic spike that left many users unable to log in on June 1.
According to a blog posted on June 6 by Coinbase software engineer Michael de Hoog, the exchange experienced a 5x traffic spike over four minutes around 16:05 PDT on June 1, around the time the price of Bitcoin (BTC) was approaching $10,000.
The software engineer said the exchange’s autoscaling was “unable to keep pace with this dramatic increase in traffic”:
“This traffic spike affected a number of our internal services, increasing latency between services. This led to process saturation of the web servers responsible for our API [application programming interface], where the number of incoming requests was greater than the number of listening processes, causing the requests to either be queued and timeout, or fail immediately. Our request error rate spiked to 50%, causing customers to experience errors when interacting with coinbase.com and our mobile apps.”
Coinbase said it redeployed the API at 16:20 to increase the number of machines dealing with this spike in traffic. Another two-minute outage followed “due to instances saturating and being marked unhealthy” before the exchange was back online.
History Of Outages
Cointelegraph reported last week that Coinbase has gone offline four times in the last three months during major Bitcoin price moves, leaving many users unable to access their portfolios.
However, Crypto Twitter later revealed the problems on the exchange went further back than that. CryptoWhale posted a chart to his 18,000+ followers on June 3 showing how Coinbase had gone offline 11 times in its history during major price moves:
After doing more research into exchanges uptimes, I’ve noticed a substandard pattern from #Coinbase.
Their exchange seems to be programmed to go “offline” anytime theres a $500+ move in #Bitcoin’s price. Over the last year, Coinbase has gone offline 11 times during larger moves. pic.twitter.com/3fNOU1QuiZ
— Whale 🐋 (@CryptoWhale) June 3, 2020
Unfortunately, outages during huge downturns or price surges hurt traders wanting to buy and sell the most. Losses can mount when Coinbase users aren’t able to access their accounts to sell their crypto, and potential profits can simply disappear when they can’t buy anything.
Will Coinbase Go Down Again?
The exchange said it was working on improvements in response to the June 1 outage. If Coinbase were to experience another traffic spike to the price of Bitcoin suddenly surging or falling, de Hoog said “pre-scaling and caching” would reduce the impact.
“Longer term we’re planning to improve our deployment process to mitigate some of the autoscaling issues we experienced,” the software engineer said.
Users Punish Coinbase For Outage by Withdrawing Record Amount of BTC
Coinbase experienced its highest single-day net BTC withdrawal in the wake of the outage incident, with even the author of “Black Swan” quitting the exchange.
Coinbase experienced the largest net outflow of Bitcoins (BTC) in ages in the wake of the recent service outage. Even Nassim Taleb has left the nest.
The June 1 outage happened during the surge of traffic as Bitcoin was trying to push through the $10,000 resistance level. Similarly, to the BitMex’ blackout a few months ago, this caused much frustration amongst its user base. A few days later, one of its most famous users, “Black Swan” author Nassim Taleb, quit the exchange.
Users Punishing Coinbase?
June 7, Coinbase users withdrew 22,000 more Bitcoins than they deposited — worth $214 million. According to Glassnode data, only once in history did Coinbase see a greater net withdrawal — July 28 2017, when the outflow was 22,500 BTC. The price at the time, however, was $2,785. Thus, denominated in dollars, that was a much more modest event, with only $63 million worth of Bitcoin withdrawn from the exchange.
Coinbase still holds close to a million Bitcoin, more than any other exchange. Thus, this latest incident may have an insignificant impact on its bottom line. Nonetheless, keeping in mind how BitMex lost its leadership position, Coinbase would be well-advised to fix its scalability woes in order not to alienate its users any further — black swans appear when you least expect them.
Coinbase Outlines Tech Plan To Help Avert Future Outages
After a traffic spike again caused severe disruption of its exchange service – and angered users – Coinbase said it’s working on technical changes aimed to provide more stability in the future.
In a blog post published late on Friday, Coinbase said it is ready to implement “a number of improvements,” which would allow the exchange’s servers to handle a sudden surge in usage – as was experienced last Monday.
“Around 16:05 PDT [23:05 UTC], the price of BTC reached USD $10,000. In connection with the rising price, we experienced a 5x traffic spike over 4 minutes. Our autoscaling was unable to keep pace with this dramatic increase in traffic,” reads the blog.
This traffic spike led to increased latency, Coinbase said, which had a domino effect on other services. With its servers becoming saturated with users trying to access and use the platform, and error reports also spiked by around 50% as visitors experienced timeouts and other server errors.
Within 20 or so minutes the firm had redeployed its API to increase the number of servers dealing with the traffic, as per the post.
After a review of the issues, Coinbase went on to list some of the changes that are currently being implemented to prevent similar traffic-spike outages from happening again:
The server “health check” system, which caused erroneous automated responses that worsened the June 1 issue, has already been updated to ensure that overloaded processes don’t get taken out of service.
Coinbase is also adjusting its systems to lower the impact of traffic spikes though “pre-scaling” – creating more server instances under heavy load. It will also rely more on caching, which has users load a stored version of pages in the browser, rather than reload full pages every time the website is visited.
“I personally think that autoscaling is great for the e-commerce industry” commented Paolo Ardoino, CTO at rival exchange Bitfinex. “Spikes in website traffic at exchanges can take place within milliseconds and place significant demands on an exchange’s capacity to maintain high availability at all times.”
Longer term, there are other changes planned too. Coinbase was light on the detail, however, saying it aims to improve its “deployment process to mitigate some of the autoscaling issues we experienced.
The exchange has something of a history of upsetting users with outages during important price moves. After online criticism reached conspiracy-theory levels after last week’s issue, Coinbase’s Justin Mart tweeted: “We do not purposely take down the site.”
CoinDesk reached out to Coinbase for more detail on the planned changes but hadn’t received a reply by press time.
Blackout Cure Needed? BTC Price Volatility Challenges Crypto Exchanges
2020 has been one of BTC’s most volatile periods, but why do cryptocurrency exchanges keep crashing under high demand?
Nobody can really predict Bitcoin’s (BTC) price volatility. But one thing is becoming painfully predictable when the price of Bitcoin suddenly lurches in one direction or another: One or more of the major cryptocurrency exchanges simply goes offline.
This leaves users powerless to prevent losses from spiraling, as they are left unable to trade or buy more positions as a hedge.
These outages have happened time and time again. Most recently, as Bitcoin started climbing toward the $10,000 mark, Coinbase went offline. At the time it happened, Cointelegraph reported that this is the fourth time in the last three months that Coinbase has shut down during major moves in the price of BTC. Additionally, Twitter user CryptoWhale pointed out that there have been no fewer than 11 Coinbase outages over the last 12 months, each one coming at a time when Bitcoin’s price had moved more than $500 in value.
The Silicon Valley-based exchange later issued a statement via its blog, clarifying that the June 3 outages were due to an issue with its API, which was seeing five times more traffic than usual. Without directly addressing the issue of the frequency of outages, the blog post stated that Coinbase is “working on reducing the impact of price-related traffic spikes though pre-scaling and caching.” Meanwhile, the exchange saw users withdrawing BTC en masse, following the incident.
A Broader Problem?
During March’s Black Thursday, BitMEX went offline for 25 minutes, subsequently blaming two separate distributed denial-of-service attacks. However, Twitter users, including the CEO of rival exchange FTX, Sam Bankman-Fried, and trader lowstrife, called foul play.
BitMEX denied the allegations, but it’s not the first time that the Seychelles-based exchange has been accused of playing dirty.
Blogger Hasu aired his suspicions that the company “weaponizes their server problems” back in 2018. This issue is also clearly set out in the form of allegations in the class-action lawsuit currently pending against BitMEX, which states: “BitMEX routinely freezing its servers — which BitMEX blames on technical glitches and limitations — to profit from moments of high volatility.”
A month after Black Thursday, the company saw a drop of 38% in its Bitcoin holdings. It is unclear whether the drop is due to users losing trust in the platform or because of the overall market sentiment, whereby an abnormally high amount of BTC is being withdrawn from exchanges. Meanwhile, BitMEX has struggled to regain the open interest lost in March, implying that it may be losing market share to its smaller rivals, such as Bybit and FTX.
Coinbase and BitMEX are the two platforms that have most often come under fire for downtime in volatile markets. However, data provider Kaiko performed an in-depth analysis of the minute-by-minute trade data for March 12 and 13, covering seven spot and six derivatives exchanges. Five of the spot venues and four of the derivatives platforms were found to have experienced some kind of issue during peak moments of volatility.
Of the spot platforms in question, only Binance and Bitstamp held up, although Binance CEO Changpeng Zhao acknowledged some “glitches on peripheral systems” in a tweet. In derivatives, Binance Futures and Huobi DM managed to ensure uninterrupted trading.
A Problem Unique To Crypto?
Contrary to the crypto sphere, there are no regular instances of the traditional stock markets going down during peak trading hours. Of course, the stock markets don’t see the same volatility as cryptocurrencies, but they do handle trading volumes that are far greater than any crypto exchange. The biggest cryptocurrency exchanges have enough experience of Bitcoin’s volatility to be able to anticipate certain kinds of peaks.
Arguably, some big exchanges also have money to invest in building infrastructure that can handle the kind of volume spikes. Coinbase has raised over half a billion dollars over its lifetime. It could be argued that BitMEX is a minnow, by comparison, having raised only $25,000 with the last seed round in 2015. However, one analyst estimates that BitMEX is raking in around $700,000 per day in fees from its derivatives trading service, which could come to over $250 million each year. Joel Edgerton, the chief operating officer of bitFlyer USA, believes that the issue is one of industry maturity, telling Cointelegraph:
“Crypto exchanges do not have the deep institutional experience that is in a traditional stock exchange. Traditional exchanges have had over 100 years to build the skills, processes and systems needed to handle the volumes they receive.”
Different exchanges appear to have different views on what could be an appropriate fix for the downtime problem. Bitfinex has recently issued a press release boasting its own performance in 2020, stating that it’s had no major incidents of downtime so far this year. The exchange points to its “obsessive interest in technical improvement” as being the reason why it has managed to achieve this.
Other exchange leaders seem to agree unanimously that the focus should be on the technology above all else. Edgerton explained that bitFlyer was built in Japan by specialists from Goldman Sachs and undergoes continual rigorous performance testing. On the matter, Catherine Coley, the CEO of Binance.US, pointed out to Cointelegraph that:
“Our infrastructure is built to regularly handle over $10 billion of daily trading activity. We have capacity for much larger volumes before our systems would become stressed.”
Stephen Stonberg, the COO of Bittrex, believes that many exchanges may be underestimating the effort and expertise involved in building an exchange that can withstand high volumes during peak times. He told Cointelegraph: “We can say with certainty that building a rock-stable exchange is harder than it looks.”
While it seems acceptable enough that the crypto industry simply isn’t developed enough to handle high volatility at peak yet, the debate still rages over whether the industry should implement circuit breakers. Meanwhile, some have accused exchanges of using their downtime as a form of a hidden circuit breaker.
Several platforms that have suffered from outages, including Robinhood and Gemini, don’t offer leveraged trading.
Furthermore, both Coinbase and Gemini are regulated in the United States. Therefore, it would be a stretch to assume that the platforms take their services offline deliberately. If anything, spot exchanges risk losing out on the fees from users who would happily dump their holdings at market rates instead of setting a limit order.
Edgerton cautiously told Cointelegraph that there could be two reasons behind the outages: “In one theory, an exchange has lost focus on the cryptocurrency community and is trying to become a financial conglomerate with a hand in every pot.” He added: “In this case, frequent volatility-caused outages are the result of not taking care of the basics.” But it may also not just be about inadequate systems, as he elaborated on another potential theory:
“Some exchanges are closer to a casino business model than an exchange one. Volatile assets like Bitcoin do not need 100x leverage, especially when marketed to retail customers. Additionally, some of these exchanges may actively avoid regulation or not clarify where their business really exists. In these circumstances, it is reasonable that people would question mysterious outages as a hidden circuit breaker that is putting the company’s interests ahead of their customers.”
Users Demand Better Systems And Transparency
This year’s volatility has been extreme, even by Bitcoin’s standards. It’s also worth remembering that there are more traders, exchanges and general interest in cryptocurrencies now than at any point in the past. So, in a way, it’s no surprise that the infrastructure of some exchanges has started to creak and groan under heavier loads.
However, transparency is also critical. In traditional markets, regulation forces transparency. In crypto, only some exchanges have chosen to be regulated in order to operate in particular markets. In the absence of regulation, it’s reasonable to expect a certain degree of transparency from exchange operators, particularly regarding how they handle costly issues such as liquidation during volatility. In a crowded market, traders will vote with their feet, so it’s up to the crypto exchanges to meet transparency demands and system stability.
Coinbase Comes Back Online Following Issues
Coinbase users were reporting issues with certain purchases, though normal function has now seemingly returned.
Coinbase users recently reported suffering difficulties on the platform. Coinbase and its trading platform, Coinbase Pro, have a history of experiencing outages and other issues during high-traffic times.
Although users reported problems early this morning, a status update from Coinbase now indicates the platform is working normally once again. “All services are back up,” the update said at 12:33 pm EST. “We are currently monitoring our systems.”
“Coinbase has disabled buying and selling of some crypto assets,” Morgan Creek Digital co-founder Anthony Pompliano tweeted earlier on Friday morning. Multiple folks on Twitter posted pictures of their difficulties when attempting USD purchases on Coinbase.
The GameStop effect has completely upended the crypto market now. There’s a big Twitter-led $btc pump happening today and Coinbase — basically the Robinhood for entry-level crypto trading — has frozen USD purchases. Just tested it myself. pic.twitter.com/xpJCQjA0uh
— Ryan Broderick (@broderick) January 29, 2021
Crypto-Twitter personality Ryan Broderick also noted issues with buying BTC via the euro. He added in a separate tweet:
“For context, a bunch of big Twitter account have put #Bitcoin in their bios. Elon Musk being the biggest. Now $BTC is up almost 15% in the last 24 hours.”
Significant chatter has arisen in recent days surrounding drama in the financial markets and on social media. GameStop’s stock, under the ticker GME, enjoyed a significant rise in price, allegedly in connections with the efforts of a trading group on social media. Popular trading platform Robinhood recently suspended buying for a number of assets, including GME.
Coinbase often suffers platform downtime and technical difficulties. Multiple outage instances occurred in 2020, including during notable Bitcoin price moves in April and May.
Coinbase Outage Caused Crypto Purchase Issues, Not An Intentional Ban
Coinbase posted an in-depth explanation of its Jan. 29 platform difficulties.
On Jan. 29, a number of Coinbase users reported issues with the platform’s buy and sell functions. The Coinbase difficulties arose just a day after Robinhood barred GameStop share purchases following surging prices on the stock. A recent blog post indicates the Coinbase difficulties arose from an outage, not a ban on trading.
“Between 4:25am and 9:31am PST on Friday, January 29, api.coinbase.com had an outage,” Coinbase said in a post on Feb. 5.
“During this time, many users experienced errors while attempting to use the Coinbase app and crypto buying, selling and trading features were intermittently available,” the post added, also noting that users of its professional exchange, Coinbase Pro, were still able to sign on to the platform.
The cause of the problems on Coinbase? Crypto price data difficulties. “It was quickly diagnosed as an issue with a Redis cluster used to store spot rates for currency conversions,” the blog post detailed.
“These spot rates are used for estimated conversions between currencies, which puts it in the critical path for a range of functionality such as displaying the value of your portfolio in your local currency.”
Essentially, certain backend processes became overloaded and went down, based on subsequent information in the post. Coinbase’s tech team tried a number of options, eventually finding the right solution, which resurrected function on the platform.
The post also dove deeper, laying out a timeline of events on the backend during the technical disruption.
“Any requests which relied on exchange rate data were failing, and users were therefore unable to access critical functionality on our site,” the post noted as a result of the issues described in the timeline of backend events.
Coinbase listed multiple changes the team has implemented for improvement going forward. Platform difficulties are not foreign to Coinbase. The outfit has experienced many connectivity problems in the past. In November 2020, Brian Armstrong, Coinbase’s CEO, tweeted about the company’s work toward fixing its issues.
Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,Coinbase Exchange Inaccessible Due,