Ultimate Resource On Solana Outages And DDoS Attacks
Solana Mainnet Back Online After Day In The Dark. Ultimate Resource On Solana Outages And DDoS Attacks
Engineers succeed in restoring service across Solana’s network of node-runners after the system was in the dark for almost 20 hours.
Solana’s blockchain validators came back online early Wednesday following a marathon session of fits and starts by engineers racing to restore service.
After being down for more than half a day, the high-speed blockchain succeeded shortly after 1:30 am Eastern time in pushing a network patch that held across a network of validators.
“And we’re back!” declared one developer in the Solana Discord Server, the site of much of the action on Tuesday and early Wednesday.
The sentiment across Solana’s ecosystem was one of relief: a high-profile outage was coming to an end.
Even so, service remained spotty immediately following the patch, according to a Twitter account run by the Solana Foundation.
The Solana validator community successfully completed a restart of Mainnet Beta after an upgrade to 1.6.25. Dapps, block explorers, and supporting systems will recover over the next several hours, at which point full functionality should be restored.
— Solana Status (@SolanaStatus) September 15, 2021
“The Solana validator community successfully completed a restart of Mainnet Beta after an upgrade to 1.6.25. Dapps, block explorers, and supporting systems will recover over the next several hours, at which point full functionality should be restored.”
Traders appeared upbeat on the news. SOL, the native token of the network, quickly popped over 3%, reclaiming some lost ground but still short of yesterday’s highs. SOL had been down 13% to the mid-$140 range at Tuesday’s lowest point.
Solana Validators Plan Network ‘Restart’ As Outage Continues
Solana developers have decided to restart the network after over seven hours of downtime.
Solana node operators are coordinating a “restart” of the high-speed blockchain Tuesday in an attempt to bring the stuttering network back online.
“The validator community elected to coordinate a restart of the network” and is preparing a “new release” that will right the frozen blockchain, according to an afternoon tweet from the Solana Foundation.
In Solana’s Discord server, developers distributed restart instructions at 3:21 p.m. Eastern time.
The outage began early Tuesday after “resource exhaustion” on the network brought blockchain validation to an hours-long halt. On-chain activity froze across Solana’s multibillion-dollar ecosystem of trading, staking and lending projects.
A source familiar with the matter told CoinDesk that the outage “impacts everything built on Solana, but the issue is the underlying [layer 1].”
Tuesday’s outage came amid booming interest in Solana, a network seeking to attract Wall Street usage whose token has been on fire since late August. Investors have flocked to Solana as a low-fee platform for decentralized finance (DeFi).
The disruption was caused by bot trading of Grape Protocol’s Tuesday token offering, founder Anatoly Yakovenko said on Twitter. It is the second time in two weeks that Solana’s mainnet cluster experienced “instability,” according to a Messari research note reviewed by CoinDesk.
A flood of transactions pushed the network past its limits early Tuesday, Solana’s status account tweeted shortly after 3 p.m. Eastern time. The activity triggered a forking that knocked multiple nodes offline. Engineers tried and failed to triage the problem, according to Solana.
It’s not unheard of for blockchain networks to experience service disruptions. Solana’s mainnet is still in beta mode; it last experienced an outage in December, another source said. Last week’s hiccup was also related to a resource exhaustion event.
“It’s good for people to understand that Solana mainnet is still clearly in beta,” said a source building one of Solana’s popular DeFi protocols interrupted by the outage. “I have faith in the Solana core contributors and validators to resolve this issue promptly.”
Engineers were rushing to push a patch to Solana node runners, multiple sources told CoinDesk. Once it’s ready and running across 66% of validators (a supermajority on the network) Solana is expected to come back online.
Until then, the ecosystem with over $11 billion in total value locked (TVL) is in wait-and-see mode. Activity remains frozen at block #96538485.
It is unclear what implications the Tuesday outage may have for Solana. One source in the developer community pointed out that some projects have time-sensitive operations, like liquidations and IDO launches. Those could become thorny the longer the network remains down.
Solana Attributes Major Outage To Denial-Of-Service Attack Targeting DEX Offering
The Solana Foundation says bots spammed the Grape Protocol IDO on Raydium with 400,000 transactions per second, bringing the network to its knees.
Solana has attributed the 17-hour outage it suffered last week to a denial-of-service attack aimed at Grape Protocol’s Sept. 14 initial DEX offering (IDO).
In a Tuesday blog post, the Solana Foundation stated that bots spammed the network as Grape launched its IDO on the Solana-based decentralized exchange (DEX) Raydium at 12:00 UTC last Tuesday.
The botting activity overwhelmed the network with a transaction load of 400,000 per second, with Solana noting that “unbounded growth of the forwarder queues and resource-heavy blocks” resulted in a number of forks being automatically proposed to the network.
The attack caused Solana’s network validators to crash after running out of memory. As a result, the network went offline for roughly 17 hours during Sept. 14 and 15.
The recovery was led by a collaboration among Solana engineers and more than 1,000 validators, with a hard fork being passed after receiving support from 80% of the network’s active stakers.
“This was a coordinated effort by the community, not only in creating a patch, but in getting 80% of the network to come to consensus.”
The foundation estimates that the network was patched, upgraded, and restored to full functionality within 18 hours of Solana going offline.
The post added that the community is still working on providing a detailed “technical post-mortem and root cause analysis report” that will be released in the coming weeks
The price of Solana (SOL) has performed bearishly since posting an all-time high of $213 on Sept. 9. Since then, SOL has pulled back by 39% to change hands for $129 at the time of writing.
The retracement followed a meteoric couple of months for SOL, with the token surging 565% since trading for $32 on July 31.
Bitcoin Flash Crashed To $5K On Pyth Network’s Data Feed
The Pyth BTC/UDC aggregate price had several sharp dips below $40,000 on Monday and the confidence levels became extremely wide.
Bitcoin briefly crashed to $5,402 (not a typo) Monday on Pyth Network oracle’s BTC/USD feed, causing liquidations to occur on a highly uncertain published price.
Per the official blog post, the unusually low price was registered between 12:21 and 12:23 UTC on Monday when the cryptocurrency traded around $43,500 on major centralized exchanges.
The Solana-based oracle network acting as a bridge between blockchains and real world data said the flash crash was caused by two of its data sources publishing a near-zero price and receiving a relatively higher weight from the network’s aggregation logic. As such, the average price crashed to lows near $5,000.
“The Pyth aggregation logic combined these prices with nine other publishers resulting in a low aggregate price with a wide confidence interval,” Pyth Network said on Wednesday.
The Pyth BTC/UDC aggregate price had several sharp dips below $40,000 on Monday and the confidence levels became extremely wide.
Confidence interval refers to the degree of uncertainty with any statistic and is often used with a margin of error. Pyth Network compares publishers’ confidence intervals in dollar terms. So, lower price implies a smaller confidence interval in dollar terms and higher weight in the aggregate.
Solana programs that rely on Pyth prices were impacted by this plunge. The effect was exacerbated due to some programs relying on the aggregate price feed without using the confidence interval price, according to the statement. “This allowed liquidations to occur even though the published price was highly uncertain,” said Pyth Network.
The flash crash has turned out to be bad optics for Pyth, with the rival Chainlink community criticizing the Solana-based oracle for data inaccuracies.
4/ While one of Pyth’s primary selling points is “data accuracy,” this is clearly one of their weakest design point in reality
Their aggregation weights sources by their “confidence threshold”, but the oracles can report any confidence threshold value they want
— ChainLinkGod.eth 2.0 (@ChainLinkGod) September 21, 2021
It also provided an opportunity for devotees of traditional centralized finance to criticize oracle-based decentralized finance as a whole.
“On-chain oracle data often has problems with corrupt feeds or corrupt data,” said Patrick Heusser, head of trading at Crypto Finance AG. “You kind of take that as a risk if you have positions open on a decentralized exchange.”
Decentralized exchanges get their data on-chain from oracles. Data distribution on-chain is slow and sometimes false data gets distributed as there aren’t sufficient quality checks performed, unlike paid data from Bloomberg or Reuters, for example, Heusser explained.
This isn’t the first issue that the Pyth Network has suffered. Connected to the Solana blockchain, which was down for 17 hours last week, Pyth also witnessed the outage.
“Solana’s recent turbulence in the form of a 17-hour outage, which meant that users were unable to process transactions, together with an eye-catching error by Pyth data network which is run by some of Wall Street’s biggest players, is certainly not to be taken lightly by the core dev team,” said Denis Vinokourov, head of research at Synergia Capital.
However, Vinokourov added that this should not be the cause of alarm for its long-term growth prospects.
“After all, it is unlikely that these issues would have manifested if not for the overwhelming growth of the network. Even Google suffers an occasional outage,” he said.
In order to avoid this from happening again, the Pyth Network said in its statement that it will be developing a suggested integration testing protocol that publishers can run to validate their software changes, making it easier to run sanity checks and validate the data.
Secondly, it will adjust the aggregation logic to properly weigh prices that span a large range of values, and lastly, it will be focusing on accurately reflecting the market through “enhanced documentation,” according to the statement.
Grape Network, The Startup That Broke Solana, Raises $1.8M
Key contributor Dean Pappas said he didn’t mean for his project’s public token sale to crash an entire ecosystem. Now VCs are following suit.
Grape Network, the project whose token sale broke Solana last week, has raised $1.8 million in total from venture capital (VC) firms and that now-infamous $GRAPE offering.
The community toolkit developer closed a $1.2 million round led by Multicoin Capital on Thursday. The venture funding comes on the heels of last week’s $600,000 public sale that was so overheated with bot-buyers that it knocked host blockchain Solana offline for nearly a full day.
“We didn’t break it out of intention,” Grape Network founder and core contributor Dean Pappas told CoinDesk in an interview. “But we were part of the reason it got broken.”
His startup has been building tools to help non-fungible token (NFT) holders prove their JPEG bona fides since a Solana hackathon in May. Now, top Solana projects like Degenerate Ape Academy and Saber use Grape, among others, perhaps explaining the excitement around its token sale.
Grapes Of Wrath
That a rather niche project like Grape could break the blockchain with the highest-claimed transaction capacity of any major project raised serious questions for Solana’s tech stack.
Project leads plan to issue a full post-mortem in the coming weeks, but early assessments from the Solana Foundation point to the $GRAPE IDO, or initial decentralized exchange (DEX) offering, as the likely culprit.
Pappas told CoinDesk he was rather unfazed by the debacle. For starters, he knew the public sale would be hot. Prospective $GRAPE buyers had signaled $8.7 million in interest for a sale capped at $600,000. And nabbing IDOs on Solana’s Raydium platform have been lucrative plays lately, Pappas said.
“The last IDO before $GRAPE went up who knows how much,” he said. “Everybody knew that this was going to be doing very well, so that’s why these bots were set up to do 400,000 transactions per second, attacking the Serum market.”
Their automated onslaught was too much for Solana; it went dark for 17 hours, nixing the Grape team’s plans to put $GRAPE, whose holders get perks in the project’s Discord server, to use. Solana was righted by morning, Pappas said.
Besides Multicoin, crypto venture firms SkyVision, Definitive, LongHash, Double Peak Group and Solana Capital also participated in the $1.2 million private token sale whose proceeds will be used to add staff.
“That’s all gonna be for team expansion,” Pappas said of the cash, highlighting a particular need for engineers.
He said Grape has at least three “future tools” planned that will help crypto-based communities – from decentralized autonomous organizations (DAOs) to NFT clubs – connect their members’ JPEG personas with internet messaging hubs, such as Discord.
“The way our system works is it validates that this person who’s talking to you on this Discord has that NFT,” Pappas said of Grape Access, the flagship product that’s already live.
Building a metaverse-meatspace verification bridge is nothing new to crypto. A handful of projects like Collab Land are at work on Ethereum. But Pappas said Grape is the biggest yet on Solana, which he claimed is a preferable blockchain for DAOs because he said it has lower fees.
Grape is two months away from launching a non-custodial tipping feature that will let community members pay each other without leaving the DAO. Little features like this are essential in building out crypto-infused online communities, Pappas said.
For now, Grape’s main utility is in ensuring Discord degens own the apes they claim to. Pappas said the five-year plan is far more “grandiose” – turning Grape into a community-builder’s toolkit that can support the backbone of any project, “all the way up to a government.”
Solana Validators, Engineers Grapple With Blockchain Slowdown On Public Call
After a bout of Thursday morning congestion, members of the high-speed blockchain’s validator community voiced concerns about the network’s reliability.
Solana’s superfast blockchain slowed to a relative crawl Thursday morning due to a mysterious congestion event that spurred debates about centralization, communication and transparency at the ecosystem’s critical stakeholder, Solana Labs.
Those tensions were on display midday after validators, peeved by slowdowns to Solana’s claimed Nasdaq-like speed, joined an impromptu roundtable.
The problem was simple: A network that usually processes more than 2,000 transactions per second was stumbling along at speeds below 500. (Ethereum processes around 15 transactions per second.)
The issues had been mostly resolved by Thursday evening, but not before about 35 community members attended a public video chat, including a CoinDesk reporter and at least two Solana Labs engineers.
Solana’s 1,000-plus validators are the computing power behind a nearly $12 billion ecosystem of lending, trading and other decentralized finance (DeFi) platforms. Only a handful of them came to Thursday’s call.
They Weren’t Happy
The 50-minute conversation started with tough questions – Could Solana benefit from a centralized observation system? – ended with a surreal one – Can the next breakpoint be staged on yachts in the Mediterranean? – and left unanswered what exactly would happen next.
“All of the apps running on top of Solana are the customers of the validators, right?” Brian Long, who operates the data site Validators.app and who convened the call, told the roundtable. “It’s just super urgent that we get this sorted out.”
Failing to do so could stunt the growth of a burgeoning financial system.
Daffy Durairaj, co-founder of popular Solana-based trading service Mango Markets, said Solana’s past outages have been pain points for the decentralized exchange’s outreach efforts.
“I’m trying to bring in these market-makers and traders and whatnot,” he said. “They just lose a lot of confidence when the nodes go down, and then they don’t know why.”
Solana suffered an 18-hour outage in September that was resolved only after a hard restart. Thursday’s event, which was not an outage, was nowhere near as severe.
Still, it provided some validators a chance to air grievances to Solana Labs engineers.
They asked why the startup that has been critical to Solana’s development and that continues to be an omnipresent force in its growth remained so tight-lipped about the goings-on of an ecosystem that is supposed to be decentralized.
“We see the GitHub changes,” said the pseudonymous Zantetsu of the validator Shinobi Systems, who commanded much of Thursday’s conversation. “But, man, it just feels like a big black hole.”
Dan Albert, executive director of the Solana Foundation, denied that the project keeps unflattering information under wraps. But he acknowledged a “conservative stance” toward sharing sensitive details that Solana’s critics could spin against the network.
“The tech is hard, and I want you guys, really invested community members, to continue to feel ownership and feel like you’re being engaged by the developers and those who are working behind the scenes,” Albert said.
Behind the scenes, there’s “a fair bit of chaos,” Will Hickey, one of Solana Labs’ newest engineers, said on the call.
“My impression in my first three weeks is that there’s a fair bit of chaos,” Hickey said. “The amount of chaos that you might expect within a startup, but maybe an amount of chaos that’s a bit scary, given how much the Solana ecosystem has grown in the last year.”
Although there was no clear reason for the slowdown, which had mostly resolved by Thursday evening, a handful of voices seemed comfortable with at least experimenting with a new layer of centralization that would aim to give Solana Labs more insight into the network.
Hickey polled the room on an “observability system.” He acknowledged such a system would be centralized by design. He said it could help the engineers pinpoint problems and prepare patches faster.
The room was skeptical but not to a fault. It would have to be a voluntary system, Zantestu said. Validators.app’s Long agreed. Solana Labs couldn’t mandate such a system without pushback.
“The crazy thing is we need them as a part of this effort, though, right? They have the insight to help us figure out where we’re going to pull the metrics from. They have the skill required to actually write the code without effing it up. And so we’ll have to pull them in at some point,” Long said.
Solana reportedly hit by DDoS Attack, But Network Remains Online
The Solana blockchain has reportedly suffered another DDoS attack that temporarily clogged the network. However, the network appeared to remain online.
Solana’s blockchain performance was reportedly hit by a distributed denial-of-service (DDoS) attack over the past 24 hours. However, the network appears to have remained online throughout.
A DDoS attack generally refers to a large number of coordinated devices, or a botnet, overwhelming a network with fake traffic to take it offline.
This wouldn’t be the first time Solana has suffered this issue, with Cointelegraph reporting in September that the network suffered a 17-hour-outage due to mass botting activity for an initial DEX offering (IDO) on Solana-based decentralized exchange platform Raydium.
The latest DDoS attack was highlighted by Solana-based nonfungible token platform Blockasset on Thursday around 3 pm UTC, after it noted that:
“We are aware tokens are taking a long time to distribute. The Solana chain is being overloaded with DDoS attacks which have clogged the network causing delays.”
Solana-focused infrastructure firm GenesysGo also reported on the matter, stating that the validator network was experiencing issues with processing transaction requests, but it called for calm as it attributed the problem to “growing pains.”
The #Solana validator network is experiencing issues this morning. TPS issues as the network works to process txn requests.
Remember, this is blockchain If your txn ID went through then it’s not lost, just pending. Growing pains is all! pic.twitter.com/2Zfagq092M
— GenesysGo.sol – Shadowy Super Coder DAO (@GenesysGo) December 9, 2021
At this stage, the nature of the incident is unclear as the Solana Foundation is yet to publicly confirm any attacks, while Status.Solana shows that the network has not suffered any outages and is fully operational at the time of writing.
However, multiple accounts on Twitter asserted that Solana suffered a global outage, with Veribit CEO Roy Murphy (and BSV proponent) stating that “Solana crashed again and is currently offline. Engineers are looking into ‘rebooting the system.’ Seriously, you can’t make this shit up!”
Earlier on Thursday, members of the r/Solana subreddit attributed the network clogging to another IDO launch on Raydium, with user u/Psilodelic writing a post titled “Why do Raydium IDOs clog the Solana network and what is being done about this?”
“My biggest concern about Solana right now is the performance impact during high volume activity connected to Raydium IDOs and launches. Literally every single performance issue in the past 6 months, including the 17-hour outage, has been a result of a launch on Raydium,” they wrote.
In response, one of the group’s moderators, u/Laine_sa, didn’t explicitly confirm if Solana’s network issues were once again Raydium-related but did note there have been concerted “stop gaps” put in place to keep Solana online since the DDoS attack from September:
“Right now there’s a stop-gap in place that prioritizes vote transactions to prevent a full crash, there are additional changes to compute limits and fees relating to this in the works but it’s not a quick fix that can be rolled out in a few weeks which is why it’s taking time. It’s being looked at however.”
Cointelegraph has reached out to multiple Solana developers for comment on the DDoS attack and will update the story if they respond.
According to data from CoinGecko, the price of Solana (SOL) has dipped 6.4% over the past 24 hours to sit at $182.79 at the time of writing. Amid a pullback across most of the top crypto assets, SOL has fallen 26.1% over the past 30 days.
Solana Hit With Another Network Incident Causing Degraded Performance
Solana reportedly went offline for four hours on Tuesday. However, Solana.Status shows there have been no issues on the network.
The Solana blockchain has suffered a third incident in just a few months that clogged the network and caused transactions to fail, with users debating whether it was caused by another distributed denial-of-service (DDoS) attack or if it was just network issues.
The scale and nature of the incident are hard to ascertain, with Coinbase, Wu Blockchain and Redditors reporting there had been an incident causing the network to slow and transactions to fail. However, Solana Labs co-founder Anatoly Yakovenko denied there had been a DDoS on this occasion.
The latest incident comes less than a month after a previous attack that saw reports the network was clogged with mass botting due to another initial DEX offering on Solana-based decentralized exchange platform Radium.
According to Wu Blockchain, the Solana network went down for around four hours in the early morning of Tuesday UTC time as a result of an apparent DDoS attack. Solana.Status shows the network has been fully operational with 100% uptime over that period.
Solana went down again at two o’clock in the morning (UTC+8) on January 4th. According to users of the official Telegram community, the attacker is suspected of using spam to conduct a DDoS attack.
— Wu Blockchain (@WuBlockchain) January 4, 2022
A post on the r/CryptoCurrency group on Reddit on Tuesday also shared several screenshots of Solana (SOL) users reporting issues with failed SOL transactions around the time of the potential DDoS and network downtime.
Coinbase also provided an incident report over the past 24 hours on the Solana network’s “degraded performance” that resulted in failed withdrawals of SOL on the crypto exchange.
“This is why you don’t use a service’s own status page to come to conclusions especially if it masquerades as a decentralized blockchain, but in reality is just a glorified database,” Redditor u/Set1Less wrote.
But other users responding to the r/CryptoCurrency post in the r/Solana community questioned the validity of the claims, with u/NiftyMufti stating:
“So instead of echoing random people’s opinions, why don’t you show the charts? A DDoS attack and downtime would have shown in the block explorers. I see no such signs. Prove me wrong. Which hours in which timezone was this supposed to have taken place?”
Yakovenko echoed as such on Twitter on Tuesday, noting that the network issues weren’t related to a DDoS and were just the “pain of getting a new runtime commercialized.”
it’s not a ddos, just pain of getting a new runtime commercialized.
$ solana ping
— transaction statistics —
8 transactions submitted, 8 transactions confirmed, 0.0% transaction loss
confirmation min/mean/max/stddev = 2119/3934/9387/2415 ms
— anat◎ly (@aeyakovenko) January 4, 2022
In a separate Twitter thread, Yakovenko also stated that the “cost model for compute is still a [little] wonky, real fix to deal with this is in 1.9, where TXs have to specify all the resources they use upfront.”
In an interview with Cointelegraph on Dec. 22, Austin Federa, head of communications at Solana Labs, said that developers are currently working to address the network’s issues, specifically in relation to improving transaction metering.
“Solana’s runtime is a new design. It doesn’t use EVM [Ethereum Virtual Machine] and a ton of innovation was done to ensure that users have the cheapest fees possible, but there’s still work to be done on the runtime,” he said.
Solana Network Faces Degraded Performance For The Second Time This Week
Solana suffered a fourth network incident in the span of a few months. Two out of the four issues happened this week.
The Solana blockchain has faced its second network performance degradation incident this week. According to Solana, this is happening because of a rise in high compute transactions.
As a result, the network capacity, which was originally advertised to be 50,000 transactions per second (TPS), was reduced to several thousand TPS. Solana cited this as the reason why users experienced failed transactions and added that its developers are already working to fix the issues.
This latest network issue came only a few days after a similar incident on Tuesday, where users experienced the same problems.
Many speculated that the Tuesday incident was due to a distributed denial-of-service (DDoS) attack, but Solana co-founder Anatoly Yakovenko responded on Twitter, saying that it’s just the “pain of getting a new runtime commercialized.”
Amid these recent events, Cyber Capital chief investment offic Justin Bons expressed his disapproval with Solana and published a series of tweets enumerating the reasons why he doesn’t support the project.
Bons claims that Solana is “consistently displaying a pattern of bad behavior” and “prioritizing attracting ignorant investors over good blockchain design.”
Bons also criticized the security of the network, mentioning that DDoS attacks are not the only concern. He said that DDoS can be combined with a 51% attack. With this, he claims that attackers can “temporarily gain proportional-staked control over the network by attacking other stakeholders.”
Yakovenko dismissed this as “exhausting nonsense,” stating, “It’s impossible to DDoS a private key.”
Last year, Solana was hit by a DDoS attack causing a similar effect and degrading the network’s performance.
Solana Labs head of communications Austin Federa said that the outage came after a number of transactions during an initial DEX offering “landed in a Solana block that took an excessive amount of compute power.”
“Compute for those kinds of transactions wasn’t properly metered by the network, and caused blocks to take much longer to process than the network expected,” Federa stated.