Twitter Dee & Twitter Dumb

[Thad]

[Grath]

[beatbandito]

[beatbandito]

that's weird, I wonder why?


(okay I guess I should say this is also a joke tweet)

[nosimpleway]

second is a parody account but as perfectly-timed and -executed jokes go it's an A+

[Metal Slime]

If the resignations are as severe as claimed, Twitter's not even gonna make it to the end of the year.

[Thad]

If the resignations are as severe as claimed, Twitter's not even gonna make it to the end of the year.

Based on that first tweet people are talking like it might not make it through the end of the week. And tomorrow's Friday.

[Mongrel]

[Grath]

If the resignations are as severe as claimed, Twitter's not even gonna make it to the end of the year.

Calling it now: Twitter is going to go offline on Sunday morning, sometime between 8am and noon Eastern. Depending on how many people Elon can beg to come back for substantially more pay, it may or may not ever come back online.

[KingRoyal]

I don't think it'll go down this weekend, but with the World Cup coming up it'll probably be done for by the end of the month, as servers and storage becomes overloaded with no one around to mitigate the effects of several million posts coming in

What a sad death for the site. It may have been a hell site at times, encouraging some of the worst impulses of people, but there was some genuine comedy and warmth there. Artist, craftspeople, musicians, journalist, writers, and a whole host of people relied on the site to make connections, promote themselves and generally just have a place to shoot the shit. It went from being run fairly steady to crashing in less than a month

But the real damage will be down to Twitter staff, who did the work of actually building and running the site then had their livelihoods destroyed a month right before the holidays began. They're now cast out into a fairly rocky job market, where several other tech companies that rely on their talent laying off tens of thousands of employees, as well. It'll be a cold winter, and while some may land on their feet, others may be seeking new life paths. There will be some recompense with severance packages and the number of lawsuits that will spring out of this based on Elon's blatant disregard for labor laws in multiple companies, but those take time to get through the courts and I'm sure he'll be looking for settlements to avoid any actual legal accountabilty

Comparatively, Twitter's upper management and execs will do fine, their contracts and severances allowing them to walk away with enough money to get out of the tech industry for good, if they have the good sense to actually do that

Elon stands to lose the most, his $44 billion going down the drain with the potential for it to reach out and destroy his holdings in the other companies he mismanages. Though at the end of the day, he'll take a hit and lose s more than all of us and our loved ones will ever seen in our lifetimes, but still be left with more wealth than all of us and our loved ones will ever seen in our lifetimes

The most damaging thing for Elon might be the hit to his reputation, finally pissing off journalists enough to stop pretending he's a real life Tony Stark and actually start digging into his wild self-mythology

[Büge]

[Thad]

Calling it now: Twitter is going to go offline on Sunday morning, sometime between 8am and noon Eastern.

First World Cup game starts at 11 AM Eastern, so if it's going to go down on Sunday I'd expect it sometime after that.

What a sad death for the site. It may have been a hell site at times, encouraging some of the worst impulses of people, but there was some genuine comedy and warmth there. Artist, craftspeople, musicians, journalist, writers, and a whole host of people relied on the site to make connections, promote themselves and generally just have a place to shoot the shit.

Yeah, I've seen guys like Brockway talk about how there's not really any alternative available for connecting with fans, potential podcast guests, community in general. That's a bummer, and for all that the open-internet advocate in me says "this is what happens when you rely on a private company's walled garden" it's not their fault they went to the site where the people were. I've got the luxury of choosing not to have a Twitter account, but people who need it to run their business don't.

And the guy who owns the site showing his ass and the people who actually run it walking out.

Not that I can think of any messageboards where that happened.

[Thad]

Resignations Roil Twitter as Elon Musk Tries Persuading Some Workers to Stay

Mr. Musk’s team also held meetings with undecided employees who are key to Twitter’s operations to try to persuade them to stay, three people said. In his pitch, Mr. Musk said that he knew how to win and that those who wanted to win should join him, one person who spoke with him said.

In one of those meetings, some employees were summoned to a conference room in the San Francisco office while others called in via videoconference. As the 5 p.m. deadline passed, some who had called in began hanging up, seemingly having decided to leave, even as Mr. Musk continued speaking, two people familiar with the meeting said.

God damn. Hanging up on Elon Musk is a memory they're going to treasure for the rest of their lives. This is a story all their friends are going to get tired of hearing.

[Mongrel]

I don't think it'll go down this weekend, but with the World Cup coming up it'll probably be done for by the end of the month, as servers and storage becomes overloaded with no one around to mitigate the effects of several million posts coming in

What a sad death for the site. It may have been a hell site at times, encouraging some of the worst impulses of people, but there was some genuine comedy and warmth there. Artist, craftspeople, musicians, journalist, writers, and a whole host of people relied on the site to make connections, promote themselves and generally just have a place to shoot the shit. It went from being run fairly steady to crashing in less than a month

But the real damage will be down to Twitter staff, who did the work of actually building and running the site then had their livelihoods destroyed a month right before the holidays began. They're now cast out into a fairly rocky job market, where several other tech companies that rely on their talent laying off tens of thousands of employees, as well. It'll be a cold winter, and while some may land on their feet, others may be seeking new life paths. There will be some recompense with severance packages and the number of lawsuits that will spring out of this based on Elon's blatant disregard for labor laws in multiple companies, but those take time to get through the courts and I'm sure he'll be looking for settlements to avoid any actual legal accountabilty

Comparatively, Twitter's upper management and execs will do fine, their contracts and severances allowing them to walk away with enough money to get out of the tech industry for good, if they have the good sense to actually do that

Elon stands to lose the most, his $44 billion going down the drain with the potential for it to reach out and destroy his holdings in the other companies he mismanages. Though at the end of the day, he'll take a hit and lose s more than all of us and our loved ones will ever seen in our lifetimes, but still be left with more wealth than all of us and our loved ones will ever seen in our lifetimes

The most damaging thing for Elon might be the hit to his reputation, finally pissing off journalists enough to stop pretending he's a real life Tony Stark and actually start digging into his wild self-mythology

I think the comment I appreciate most is "This wasn't just a hell site. It was a hell home."

[Mongrel]

Fun things happening along the way:

- The team that manages access has all resigned so any other employees who've resigned have not had their access to internal systems revoked
- Even the PAYROLL team is gone. Like. I don't even. I can't think of a division on a company that's got more granite-level job protections than payroll. I don't even know if this means a bunch of people are going to continue getting paid money they should not, or what's going to happen to the severance packages or WHAT.
- Metaverse is still absolutely dead, lmao.

[Mongrel]

Who knows how long it'll stay up, but this thread listing What Can Break is pretty comprehensive, as well as plain fascinating.


Better format (and one more likely to survive the end of the day) https://threadreaderapp.com/thread/1593 ... 78592.html

EDIT: Not sure if TRA goes blank if Twitter dies, so text:
⇳ Click to Expand Summary

I've seen a lot of people asking "why does everyone think Twitter is doomed?"

As an SRE and sysadmin with 10+ years of industry experience, I wanted to write up a few scenarios that are real threats to the integrity of the bird site over the coming weeks.
For context, I have seen some variant of every one of these problems pose a serious threat to a billion-user application. I've even caused a couple of the more technical ones. I've been involved with triaging or fixing even more.

1) Random hard drive fills up. You have no idea how common it is for a single hosed box to cause cascading failures across systems, even well-engineered fault-tolerant ones with active maintenance. Where's the box? What's filling it up? Who will figure that out?

2) Physical issue with the network takes down a DC. I gather Twitter is primarily on-prem, and I've seen what happens when a tree knocks out a critical fiber line during a big news event.

3) Bad code push takes the site down. Preventing this was my day job, and I can tell you that it's one of the scariest scenarios for any SRE team, much less a completely understaffed and burnt-out one.

4) Bad code push takes the site down *in a way that also fucks up the ability to push new code*. This is the absolute nightmare scenario for teams like mine. When something like this happens, it's all hands on deck. Without deep systems understanding, you might never get it back.

5) Mystery SEV. Suddenly, the site goes dark. The dashboard is red. Everything seems fucked. There's no indication why. You need to call in the big guns. Teams with names that end in Foundation. Who are they? How do you call them?

6) Database is fucked. It's a big one. Everything is on fire. Who's the expert for this one?

7) Someone, say, entirely hypothetically, @wongmjane, finds a critical security vulnerability in your prod iOS app. You need to fast-track a fix, *stat*. You have a team of experts who know how to navigate Apple's Kafkaesque bureaucracy for app updates, right? I sure hope you do.

8) Someone notices that it's possible to read anyone else's DMs by loading up a particular URL. This is a SEV1, massive, all-hands-on-deck, critical issue. You need people who understand deeply how your privacy abstractions work, and how to fix them.

9) The site goes dark at 4am. The oncalls have no idea what's wrong. You *need* an IMOC (Incident Manager On Call) who knows who to wake up, why, and how. Someone who understands your systems, can synthesize information at lightning speed, and coordinate a recovery effort.

10) The system you use to *find other systems* internally goes down. None of your systems can talk to each other. The site, and all your tools, immediately fail. The tools you need to revert the breaking change are all FUCKED. Can you figure this one out with a skeleton team?

11) It's 5pm on a Friday. The dashboards all go red at once. The web fleet is seeing cascading reboots. The disks have been filling up since Wednesday. There were hundreds of code changes across multiple interlocking systems on Wednesday. Revert any of them at your own risk...

12) Oh shit. You reverted one of them. Now every locked account's tweets are visible to everyone. People might literally get murdered with machetes over their posts. That's not a hypothetical. It's now 9pm. The site is fucked. Who are you going to call?

13) The system that ensures server changes are safe to push to prod is failing. You have, say, 30000 tests that *must* run to ensure privacy/security/compliance/reliability. One of the tests is causing the failures. Can you find it? Also it's the World Cup. Also the site is down.

14) A user in the Phillipines is about to post CEI to the platform. You *cannot* leave that content up. Do you have your employees with relationships with PH law enforcement? Do you have your content moderation systems working? Do you have your moderators?

15) The FBI wants to inspect the contents of the DMs of someone they think is about to commit 9/11 2: Atomic Boogaloo. Do you have a system to grant them access? Do you refuse them access? How do you know it's really them?

16) You grant them access. Now someone from a country known for horrific human rights violations is knocking. They have an official-looking subpoena. Do you let them see a dissident's DMs? Can you articulate why? You might need to, in a very official court somewhere in Europe.

17) Another country is telling you that they want all of your data on their users stored on servers in their country. Do you have policy experts for that country? Do you have a lot of *very* motivated lawyers? Do you have an infra eng who knows how to partition your data just so?

18) GDPR. You're found in violation. It took a team of 100s of engineers, lawyers, policy experts, designers, and managers months of "hardcore engineering" to be in compliance in the first place. Can you get back? I assure you, not doing so will cost more than an org's headcount.

19) Once a day, every day, at 12:13am, a specific service in your data pipeline slows to an absolute crawl. It doesn't seem to be causing any issues, but you're a bit concerned as it seems to be getting worse. Do you assign an SRE to take a look? Do you have any left?

20) The service you use to discover other services is working fine, but one of your best engineers does some calculations and realizes it won't scale to more users and more services, and (hypothetically) you want to build a super-app called X. Do you rewrite? What do?

21) You decide to rewrite. 8 months later (lol) your new system is ready to take on its first users. Who's coordinating the migration? Do they *really* understand complex systems? Are they good with people? Can they execute? Do they have the domain knowledge they need?

22) You just hired a great-seeming engineering director from Microsoft for a core org. Slowly, their org's productivity slows, and attrition climbs *way* up. The director swears everything is fine. If you fire the director, one of your VPs suddenly has like 18 reports. What do?

23) An engineer just kicked off a command to reboot the fleet. Oops, they didn't use --slow. Now all of your caches are empty. All of them. Every request goes straight to DB. The DBs all get overloaded instantly, some start to OOM and reboot loop... How do you refill the cache?

24) World Cup. It is *the* defining event. We used to have watch parties for the traffic charts. The amount of traffic your site gets in one week is mind-blowing. It's in huge bursts. It tests *every* system you have to its limits. If one breaks, hope it doesn't cascade. It will.

25) New Year's Eve, USA East Coast. Every year. I remember sitting outside the office, fireworks exploding in the distance, frantically calling the video oncall. Everyone posts videos of their fireworks. *Everyone*. It will fill up disks and test your bandwidth to the very limit.

26) I've said it before, but... CEI. If you mishandle it, if your policy people and lawyers are not top of the fucking line, you *will* get yanked in front of Congress, in front of judges, into the evening news, places you don't want to be if you're running a social media company

27) Physical security of your offices. Security guards told me they keep *long* lists of crazies, commit them to memory. People want to fucking kill Zuck. Like ritual murder in the bathtub shit. They show up at the office *all the time*. Is your security team staffed and ready?

28) Genocide. People use your platform to orchestrate mass murder, the machetes in churches kind. And fast. Lightning fucking fast. You need to be prepared *before*. If you don't have a team who knows how to detect and stop this ASAP, your ass is getting dragged to The Hague

29) Rebellion. Millions of people will use your platform to orchestrate rebellion against their government. Do you use the tools for #28 to stop them? Do you let it ride? How do you decide? What if you let it ride and the same thing happens next week in a country you really like?

30) Bus Factor. Say you have 3 senior+ level SREs left in your Core Services org. They are absolutely indispensible, for reasons you can infer from above. How do you keep them all alive? Can they be on the same plane? What's the contingency plan if they all kick it anyway?

31) Invaders. A single box in your datacenter is mistakenly connected to the public Internet and forgotten for years (this really, really, really does happen, I promise). Someone pops the box. They're in. How do you detect it? What do you do once you do?

32) Invaders: The Quiet Ones. They're in your network. They're just watching, and waiting. Not doing anything. I promise you, a great security org can detect even this. If you don't have a great one left... What damage can be done by observation? Credit card data? Passwords? DMs?

33) Invaders: State Actors. The CCP just gained access. If they have their way, they're here to fucking *stay*. How will your security team find out? How will they find and eradicate backdoors? How will you protect user's DMs and private tweets? If you don't, people will die.

34) Invaders: The Chaotic Ones. They're here to do some fucking damage. They could delete data, reboot the cache fleet and take down the site for weeks, post nuclear threats as the POTUS... You better have a big, talented, experienced security team if you want to be ready.

35) On the subject of Invaders... How's your corporate IT security? It's easy to just think about the production fleet, but what if an eng's laptop is stolen from their Camry? Can you detect that before it's reported? Can you remotely lock & wipe? Invalidate their keys?

36) And again, how's that physical security team? Someone will absolutely try to plug a Raspberry Pi into your corp network. 100%. They'll try to spoof the wifi. Mics in the executive offices. 1960s spy movie shit. I'm not joking.

37) Content moderation. You need 3 things: a *giant* team of people checking reports 24/7, another team working on tooling to help that team, and regular psychiatry appointments for the first team. Not kidding, again. Humanity is DARK. Your moderators can and will commit suicide.

38) Oops! You didn't hire a content moderation team. Your site is full of very nasty stuff. Everyone leaves because it's so unpleasant, or (worse for you personally) you get dragged into court for breaking all kinds of decency, piracy, and privacy/harassment laws.

39) Oops! You didn't hire a team to build tooling for your content moderators. They are completely overwhelmed by the literally millions of reports. They burn out, you can't replace them fast enough, and #38 happens anyway

40) Oops! You didn't pay up for psych appointments for your moderators. Best case, they leave as husks of their former selves. Worst case, suicide. You're in the news/courtroom. (Ok, or worst-er case, one of them shows up to work with a gun. How's that physical security again?)

41) Speed: Backend requests. Some network change bumps the RX rate. Suddenly it takes a couple dozen milliseconds longer to load the feed. Ever seen the statistics for what happens to retention rate? It's mind-boggling. Now imagine it getting slightly worse, every day

42) Speed: Scrolling. You wouldn't believe how many tens of thousands of people will stop using your app, forever, if some Android change adds the slightest lag to scroll times. Do you have client perf experts left? What if the problem keeps getting worse? How do you debug it?

43) Speed: Cold start. If you add 5 seconds to the time it takes to start your app from nothing, you will TANK your onboarding rate. To an unbelievable degree. I mean, push it into the fucking ground. They'll never try the app, ever. Have perf experts on all client platforms?

44) Speed: Hot start. Some Android change makes it where loading the app takes 1 second longer. Suddenly, 25,000 fewer people are using your app every day. Those numbers will start to grow. I know it seems ridiculous, but this shit is REAL. How do you track start times? Do you?
(I know that some of these don't seem like immediate existential threats to the product, but trust me - to varying degrees, *any* one of these problems can be the difference between the app succeeding or failing. What might seem like an insignificant problem can snowball. Fast.)
(And the sweet Lord help you if you have to deal with more than one of these problems at a time. And LORD help you if it's while staffing a global communications utility with a skeleton crew after a giant loss of technical tribal knowledge.)

45) DATA LOSS. Holy shit, data loss. Do the people who understand your backups still work there? Hot backups? Cold backups? Data redundancy between regions? If you lose user data, it's game over. Trust is gone, permanently. Fire, nukes, floods, humidity, errors, drive defects...

46) Data loss again. Do you have read-only backups? A bad actor could absolutely try to wipe all your disks. Can they damage your tape drives from software? How long will it take to restore the entire site if everything gets wiped? (Hint: Probably weeks. Best case.)

47) Deletion. You need to know, for sure, that data is deleted from the site. This is far more important than you think. If a post is deleted, is it wiped from cache? What happens to that post in read-only cold storage? If you fuck this up, you're 100% gonna end up in court.

48) Employee burnout. I know this seems silly in a vacuum, but trust me, this shit is real. Look up the HBR series on it. You will lose critical employees. People you can't just buy back. No amount of money. People you can't afford to lose. Not lazy people. Good ones. Great ones.

49) Governmental interference. What if Brazil decides to cut you off? The US? Saudi Arabia? Do you have policy people, lawyers, lobbyists, ex-diplomats? Who will you appease? What about countries with conflicting interests? Say you can't make them all happy. Who do you choose?

50) Replication. Fuck. Um. You have, say... 5 primary regions. Each region has a copy of all mission-critical data. One day, some eng realizes that some data in A is different in B. This is *apocalyptically* bad. Which region is correct? How do you decide? How do you fix it?

51) Employee account privileges / service accounts. Can every employee reboot every server in prod? Should a help desk employee be able to see credit card details? How do you give access to only certain commands or boxes? Automated users? Can someone hijack an automated user?

52) DNS! If you fuck up DNS settings somehow, your entire *everything* can go dark. And I mean everything. Internal tools, the entire website, the literal DOORS TO YOUR OFFICE (that's not hypothetical). And if your DNS registration lapses, someone can steal your site! Forever!

53) Certificates. The magical lil "s" in "https" that keeps Bobby Joe at Peets from stealing your credit card details. If your website cert registration lapses, you would not BELIEVE the pain you're in for. Spoofing, services breaking, people completely unable to load the site...

54) Tests! Nope, no joke. If someone isn't keeping a close eye on your unit and integration and e2e and canary test suite, you can: Break every build and never ship. Push horrible, possibly illegal, bugs to the site. The worst is if all the tests *silently, incorrectly, pass*...

55) Payments. There are SO many laws, regulations, and policies around payments that you *absolutely must* be in compliance with. Any slip-up can get your ass yanked from the banking system. No, doggie coins won't cut it. Ads, verification, whatever - your revenue is fucking GONE

56) Spam! Spam is an *existential* threat. You need automated systems, AI-based clustering, entire teams doing manual review, and some *extremely* smart, driven, and creative engineers adept in adversarial thinking because you absolutely must be one step ahead at all times
NSFL - For those asking, CEI is the acronym for child sexual/abuse imagery :(

Acronyms:

SRE: site reliability engineer
CEI: child sexual/abuse imagery :(
GDPR: EU privacy regulation
DNS: domain name resolution (website name -> ip address)
SEV: "site event" (although really no one knows what it stands for. think... "bad thing happened")
DC: datacenter

Hey everybody! I'm so overwhelmed by all of your kind words and thoughtful questions. I can't keep up with it all!
There are a lot more scenarios we've come up with in the comments, hopefully I'll get a chance tomorrow to write them up as well.

It's almost 4am where I am this week, so I'm going to have to log off soon! Thank you again for all the kind words, it means so much.

[Mongrel]

Going to just post text, since I assume this, and several other threads, will shortly become a *lot* emptier.

everyone keeps speculating that Musk is on cocaine, but this is VC world. he is on some Russian research chemical from the 1960s which was recommended to them by a fascist catboy Urbit engineer on seventeen different kinds of crank, two hallucinogens, and eight benzodiazepenes

[Mongrel]

I was sort of half-wondering what we were going to do for a new subforum name. xD :chefkiss:

Incidentally.

@revhowardarson
coked-up neoreactionary morons crashed through the doors of the Twitter boardroom, imagining that the control panel that runs the Cathedral is in there. now they're ripping the copper wires out of the walls, imagining that one of them is connected to the reason people hate them

"we'll be greeted as liberators as soon as we can find the shadowban off switch," david sacks cries in a nasal tenor, railing a huge line of 2-CB and phenylpiracetam off of a rack of dying servers

(If you're not familiar or don't recall, The Cathedral is sort of a conspiracy theory/metaphor that Curtis Garvin aka Mencius Moldbug, cooked up to explain why the NYT won't appoint his boss, Peter Thiel, as Emperor For Life.)

[Mongrel]

It's an NYT editorial, yeah, but it's also (epidemiologist, data scientist, and enormous corvid nerd) Dr. Carl Bergstrom: Twitter Was Influential in the Pandemic. Are We Better for It? Probably nothing anyone here wouldn't really know already, but it's nice to kind of get it all in a tidy retrospect article.

tl:dr, To some extent yes, but more 'no' than 'yes'. So, yes Betteridge's law applies (more or less) which - considering he's actually cited it - I have no doubt was cheekily intentional.

[sei]

corvid nerd

global pandemic finally caw-se for concern