Something I find annoying is that being effective at SEO means being in a constant war with people whose literal job it is to be good at SEO to trap me in useless crap.

Seconding the FOSS advice from the perspective of a fellow learner.

I’m a scientist first and foremost, so I’m learning programming on the side. A lot of code that’s written by scientists is pretty grim, so attempting to understand and contribute to FOSS projects has been useful in understanding how a complex project is organised, and how to read code as well as write it.

Contributing can be pretty small, even opening a git issue for a problem, or adding some info to an existing issue. You won’t be able to just dive in and start solving problems all over, and it can feel overwhelming to try as a relative beginner, but it massively improved my skills.

I need to add that to my quotes book, it’s great

That’s fun, I’m stealing that

The data are stored, so it’s not a live-feed problem. It is an inordinate amount of data that’s stored though. I don’t actually understand this well enough to explain it well, so I’m going to quote from a book [1]. Apologies for wall of text.

“Serial femtosecond crystallography [(SFX)] experiments produce mountains of data that require [Free Electron Laser (FEL)] facilities to provide many petabytes of storage space and large compute clusters for timely processing of user data. The route to reach the summit of the data mountain requires peak finding, indexing, integration, refinement, and phasing.” […]

"The main reason for [steep increase in data volumes] is simple statistics. Systematic rotation of a single crystal allows all the Bragg peaks, required for structure determination, to be swept through and recorded. Serial collection is a rather inefficient way of measuring all these Bragg peak intensities because each snapshot is from a randomly oriented crystal, and there are no systematic relationships between successive crystal orientations. […]

Consider a game of picking a card from a deck of all 52 cards until all the cards in the deck have been seen. The rotation method could be considered as analogous to picking a card from the top of the deck, looking at it and then throwing it away before picking the next, i.e., sampling without replacement. In this analogy, the faces of the cards represent crystal orientations or Bragg reflections. Only 52 turns are required to see all the cards in this case. Serial collection is akin to randomly picking a card and then putting the card back in the deck before choosing the next card, i.e., sampling with replacement (Fig. 7.1 bottom). How many cards are needed to be drawn before all 52 have been seen? Intuitively, we can see that there is no guarantee that all cards will ever be observed. However, statistically speaking, the expected number of turns to complete the task, c, is given by: where n is the total number of cards. For large n, c converges to n*log(n). That is, for n = 52, it can reasonably be expected that all 52 cards will be observed only after about 236 turns! The problem is further exacerbated because a fraction of the images obtained in an SFX experiment will be blank because the X-ray pulse did not hit a crystal. This fraction varies depending on the sample preparation and delivery methods (see Chaps. 3–5), but is often higher than 60%. The random orientation of crystals and the random picking of this orientation on every measurement represent the primary reasons why SFX data volumes are inherently larger than rotation series data.

The second reason why SFX data volumes are so high is the high variability of many experimental parameters. [There is some randomness in the X-ray pulses themselves]. There may also be a wide variability in the crystals: their size, shape, crystalline order, and even their crystal structure. In effect, each frame in an SFX experiment is from a completely separate experiment to the others."

“The Realities of Experimental Data” "The aim of hit finding in SFX is to determine whether the snapshot contains Bragg spots or not. All the later processing stages are based on Bragg spots, and so frames which do not contain any of them are useless, at least as far as crystallographic data processing is concerned. Conceptually, hit finding seems trivial. However, in practice it can be challenging.

“In an ideal case shown in Fig. 7.5a, the peaks are intense and there is no background noise. In this case, even a simple thresholding algorithm can locate the peaks. Unfortunately, real life is not so simple”

It’s very cool, I wish I knew more about this. A figure I found for approximate data rate is 5GB/s per instrument. I think that’s for the European XFELS.

Citation: [1]: Yoon, C.H., White, T.A. (2018). Climbing the Data Mountain: Processing of SFX Data. In: Boutet, S., Fromme, P., Hunter, M. (eds) X-ray Free Electron Lasers. Springer, Cham. https://doi.org/10.1007/978-3-030-00551-1_7

Unfortunately no. I don’t know any research scientists who even make 6 figures. You’re lucky to break even 50k if you’re in academia. Working in industry gets you better pay, but not by too much. This is true even in big pharma, at least on the biochemical/biomedical research front. Perhaps non-research roles are where the big bucks are.

He doesn’t directly control anything with C++ — it’s just the data processing. The gist of X-ray Crystallography is that we can shoot some X-rays at a crystallised protein, that will scatter the X-rays due to diffraction, then we can take the diffraction pattern formed and do some mathemagic to figure out the electron density of the crystallised protein and from there, work out the protein’s structure

C++ helps with the mathemagic part of that, especially because by “high throughput”, I mean that the research facility has a particle accelerator that’s over 1km long, which cost multiple billions because it can shoot super bright X-rays at a rate of up to 27,000 per second. It’s the kind of place that’s used by many research groups, and you have to apply for “beam time”. The sample is piped in front of the beam and the result is thousands of diffraction patterns that need to be matched to particular crystals. That’s where the challenge comes in.

I am probably explaining this badly because it’s pretty cutting edge stuff that’s adjacent to what I know, but I know some of the software used is called CrystFEL. My understanding is that learning C++ was necessary for extending or modifying existing software tools, and for troubleshooting anomalous results.

A friend of mine whose research group works on high throughout X-ray Crystallography had to learn C++ for his work, and he says that it was like “wrangling an unhappy horse”.

Man, I hate people like you, because I am constantly reminded of how I really need to get round to checking out Searx, but I never get round to it.

(I am joking, because the only person who is putting pressure on me about this is me, and that’s because finding time to do this is something I want to do. Thank you for reminding me, even if I will keep procrastinating)

The thing I’m concerned about is how little non-programmers know. I think that much of the world went “oh, GenZ are digital natives, that means they’ll know their way around computers naturally” when if anything, being “digital natives” is part of the problem. But like my original comment said, I attribute a lot of blame to Microsoft’s impact on IT education.

I can’t speak much on how much programmers tend to know, because I am a biochemist who started getting into programming when studying bioinformatics, and then I’ve continued dabbling as a hobbyist. I like to joke that I’m a better programmer than the vast majority of biochemists, and that’s concerning, because I’m a mediocre programmer (at best).

Verbaceous is a great word. I’m adding it onto my “favourite words” list ,(even if it isn’t technically a word "

Reflecting on my IT education in school, it feels like it was mostly learning to use Microsoft Office. Reflecting on it makes me horrified, because I feel like we’re heading for a period where only a select few have tech skills and the skills gap we already see is going to get way worse. That’s what intense lobbying from Microsoft will get you

Vampire The Masquerade: Bloodlines had a patch for it that made it way more stable (and also added back in a bunch of cut content).

Way back, my partner played Watchdogs at launch and the stuttering was awful, and it was basically unplayable. Some random person made a patch that fixed most of the problems and made the game look closer to what it did at E3.

Random nerds on the internet are my favourite people

I had to do it for the first time last year and I was slightly giddy from the novelty of it.

Once upon a time, a thing happened. And then there was a facsimile of narrative conflict, but everything worked out in the end, because that’s how all the short stories by LLMs seem to work.

This is an excellent comment, thanks for writing this up

As a gamer, I was anxious about switching to Linux as my daily driver, but I needed to fully immerse myself to improve at Linux, and I’ve been pleasantly surprised by how few gaming related problems I’ve had.

Regex feels distinctly eldritch to me. Like, a lot of computing knowledge feels like magic, but regex feels like the kind of magic you get by consorting with dark forces

Now I’m thinking about an ex-programmer supervillain who does this as her big foray into supervillainy

“vendored my library”

I’m unfamiliar with this phrase, are you able to explain what it means (or point me towards an explanation)? Is it relating to forking?