{ "version": "https://jsonfeed.org/version/1", "title": "Murto Hilali - Compbio, tech, proteins, and more.", "home_page_url": "https://murto.co/", "feed_url": "https://murto.co/feed.json", "description": "My digital garden & portfolio. I write about computational biology, tech, proteins, and more.", "icon": "https://murto.co/favicon.png", "favicon": "https://murto.co/favicon.ico", "expired": false, "author": { "name": "Murto Hilali", "url": "https://murto.co/", "avatar": "https://murto.co/img/author/website-headshot.jpeg" }, "items": [ { "id": "https://murto.co/acarbose-alternative-drug-discovery/", "title": "How I Found 71 Natural Alternatives to a $200 Million Drug", "summary": "A step-by-step drug discovery and AI virtual screening project done from my laptop", "content_text": "By Author — Not random images either, there’s a chance you find the compounds in all of these organisms.Table of Contents Exploratory Data Analysis for COCONUT — Profiling ~100K natural products to understand drug-likeness, physicochemical space, and screening viability. Chemical Similarity Search — Fingerprint-based filtering (ECFP, MACCS, pharmacophore, USRCAT) to reduce the database to high-similarity candidates. Protein Model Selection, Cleanup, and Preprocessing — Selecting acarbose-bound crystal structures and preparing them for docking. Protein–Ligand Docking & Assessment — Running GNINA-based virtual screening and scoring with CNN-VS. Lead Validation and Controls — Decoys, redocking replicates, statistical filtering, and stability analysis to remove false positives. Candidate Overview & Key Takeaways — Final shortlist by target protein and reflections on the workflow.Flavonoids are a large, diverse group of secondary plant metabolites that play a role in pigmentation, UV protection, and insecticidal activity. Specifically, some anthocyanins (a subclass) can inhibit insect digestive enzymes, preventing nutrient uptake and development, serving as a defence mechanism against predators.Obviously, there’s potential here for bug Ozempic (or insecticides, you pick). But research on Allium mongolicum flowers has also revealed that their flavonoids may inhibit human digestive enzymes, including our very own starch-metabolizing α-glucosidase. These compounds could be used to treat diabetes, as inhibiting starch breakdown reduces postprandial blood glucose levels (Li et al., 2025).Initial docking analysis revealed that A. mongolicum-derived isoquercetin (already in clinical trials) can inhibit α-glucosidase (AGI) activity with an effect similar to that of acarbose , a common AGI derived from Actinoplanes bacteria. Chemical similarity search revealed that troxerutin (aka vitamin P4, a flavonoid you can buy over the counter) shares some substructural motifs with isoquercetin, and its docking analysis showed similar binding affinity.Docking results from DiffDock-Pocket. Note : troxerutin’s slightly higher affinity may be attributed to steric interactions introduced by the moieties found on the larger molecule.I wanted to take this approach and extend it to a larger database of natural compounds to find other biochemicals that could hypothetically outperform acarbose — after all, the drug is projected to have a $220 million global market by 2035.Here’s my overall process: Exploratory data analysis of COCONUT, a natural products database. Chemical similarity search of acarbose against COCONUT to find structurally similar NPs. Protein model selection, cleanup, and preprocessing. Protein-ligand docking and assessment , where I run a virtual screen against human α-glucosidases. Lead validation and controls to identify any false positives. You can see all the code (in progress) here. Workflow diagram1. Exploratory data analysis for COCONUTIn this section, I’ll take a look at: Distributions of some pre-existing features in the database. The spread of some derived ratios. There are a little over 100,000 natural products in the COCONUT database. I started by doing some simple feature visualization: Total atom count : all atoms, including hydrogen. Heavy atom count : any non-hydrogen atom. Molecular weight : in grams per mole ALogP : A drug discovery metric that estimates molecular lipophilicity. Topological polar surface area : The surface sum of all polar atoms, also a measure for drug bioavailability. Rotatable bond count : single bonds that allow for free rotation, generally a measure of flexibility/rigidity. HBAs/HBDs : Polarity metrics — we typically want a particular balance of the two, with more HBAs than HBDs in drug-like molecules. Lipinski Rule of Five violations : the RO5 is a rule of thumb for evaluating drug-likeness based on a set of molecular properties. However, it is worth noting that several drugs do not have these characteristics — in other words, it is not the be-all and end-all. Those properties are: 5 or fewer HBDs No more than 10 HBAs Molecular mass under 500 Da ClogP < 5 Aromatic rings count : Aromatic rings add planarity to a molecule. QED druglikeness : A drug likeness measurement that focuses on 8 features (all listed attributes other than RO5 and atom counts). NP likeness : Natural product likeness, measures a molecule’s similarity to known natural products.For a closer look at some derived ratios, I looked at the following:Research indicates that a TPSA/MW ratio >= 0.2 is ideal for good solubility; approximately half of the dataset meets this criterion (Whitty et al., 2018).This metric stems from two findings: Fsp3 (the fraction of sp3 carbons over total carbons) increases as drugs move through the discovery pipeline (Lovering et al., 2009). Molecules with more than 3 rings tend to have poor solubility, high lipophilicity, and high promiscuity (i.e., off-target binding) (Ritchie et al., 2011). Here, I take the ratio of both in the dataset. Two populations emerge: many NPs with no saturated carbons and a smaller group with some balance.Takeaways : COCONUT offers a relatively diverse search space There appears to be drug-like potential amongst several NPs2. Chemical similarity searchChemical similarity is generally determined by generating chemical fingerprints for a molecule, fixed-size identifiers that capture information about atomic structure and chemistry. Several fingerprinting protocols exist, each with its own strengths and weaknesses. Here are the ones I chose and why: ECFP. Extended-Connectivity Fingerprints represent molecular structures and are widely used in cheminformatics. They enable substructure and similarity searching but are also used for quantitative structure-activity relationship (QSAR) modelling. If you’re interested in molecular activity, these are a good pick. Pharmacophoric features. These fingerprints encode data on chemical features typically involved in pharmacological actions, such as hydrophobicity, charge, aromaticity, etc. This makes them useful when we’re looking for chemicals with properties similar to those of acarbose. MACCS keys. Short for Molecular Access System, MACCS are one of the simpler fingerprinting methods. They encode the answers to established T/F questions about chemical structures, such as the ring size, ion presence, or oxygen count. They can be generated quite quickly and are relatively interpretable. Ultrafast Shape Recognition. USR is the only 3D method on this list. It generates a vector of shape descriptors for a set of conformers of a given molecule. USRCAT, which I am using more specifically, also encodes pharmacological features. By generating fingerprints for acarbose, we can use Tanimoto similarity to assess the degree of agreement between our database compounds and the lead target. USR is the exception; we use inverse Manhattan distance instead.To trim the dataset down, I used the following cutoffs for Tanimoto similarity: ECFP: 0.4. Research seems to show enrichments of similarly active compounds beyond a TC of 0.4 with minimal false positives. MACCS: 0.7. See above. Pharmacophoric features: 0.8. A slightly more arbitrary cutoff, but I elected to be stricter since ECFP and MACCS are fairly generous. USRCAT: top 1%. In virtual screening workflows and decoy comparison benchmarks, the top 1% of candidates are selected. This gave me a dataset of 10,341 high-similarity candidate molecules to explore. Ligands were generated using RDKit and converted to .pdbqt using OpenBabel.3. Protein model selection, cleanup, and preprocessingAcarbose is a bit of a rolling stone, it can’t be tied down — by which I mean it interacts with multiple proteins. To be thorough, I wanted to explore this virtual screening process across all of its protein targets at once. In hindsight, this made it much harder to keep track of (I’ll take it one target at a time from now on).I made my selections based on protein targets listed in DrugBank and the resolution of the corresponding protein model. I also specifically selected PDB entries of the target protein _ in complex with acarbose_ so I’d a) have a defined binding pocket and b) have an experimental reference point for future validations. Finally, all proteins were put through PDB-redo before being converted to .pdbqt for docking: 2QMJ : The N-terminal subunit of human maltase-glucoamylase in complex with acarbose. As you can see from the PDB page, there’s room for growth in model quality, but PDB-redo led to some minor improvements. (Gene: MGAM). 3BAJ : Human pancreatic alpha-amylase in complex with nitrate and acarbose. PDB-redo led to more significant improvements here. (Gene: AMY2A). 5NN8 : Human lysosomal acid-alpha-glucosidase in complex with acarbose (Gene: GAA).Below, acarbose is designated in light green with binding residues (within 4 Å of acarbose) in red. All other observed elements are ions and ligands, which are removed for docking.Note: one of the highlighted light green ligands in 5NN8 is an acarbose-derived trisaccharide; the other binding pocket (containing true alpha-acarbose) is used for docking configuration — Images generated via The Protein Imager.4. Protein-ligand docking & assessmentFigures generated using ChimeraX. Can you tell I tried (unsuccessfully) to create Goodsell-like graphics?For protein-ligand docking, I usedGNINA, an open-source AutoDock Vina fork that uses CNNs to score ligand poses.Since we know where acarbose is supposed to bind (the protein models include acarbose), we can guide the docking process by setting an autobox (basically a mini 3D search space) around its coordinates. In the diagrams above, I’ve indicated the autobox GNINA draws around the ligands (4 Å outward from the farthest corners of the molecule) in red.While it may be the most computationally expensive step, this stage of the project was actually the easiest. I ran GNINA on an HPC cluster in parallel fashion to save time — although GNINA wasn’t designed specifically for high-throughput virtual screening1, this step was completed in approximately one day.5. Lead validation and controlsIn this section, I go over: Filtering through the results of our initial docking run. Analyzing the results of decoy docking tests. Compare lead ligand performance across replicates. GNINA generates several metrics for binding poses , including CNN pose score (the AI’s assessment for score quality) and CNN binding affinity (the AI-predicted kcal/mol). Benchmarking studies with GNINA have indicated that the product of these two values, CNN_VS (virtual screening), can be used to identify potentially active ligands when it exceeds 6.30.Horizontal lines represent the CNN-VS threshold of 6.30.Taking the CNN_VS of all ligands docked against each protein, 71 ligands passed the threshold.Okay, the headline promise-payoff is complete—but we’re not done yet. After collecting a set of leads, I needed to validate some of the findings. There were a few ways I went about doing this, with each control telling me something different: Molecular decoys. For each lead, I used LUDe to generate physicochemically similar molecules for docking against the protein receptors. Decoy molecules tell us whether there’s something unique about a lead or if any molecule with a similar molecular weight, log P, polarity, etc., would perform just as well. Redocking with variant seeds. By using different seeds, we can check whether the results from our first run are consistent or a fluke — in other words, we need to run replicates. Structural interaction fingerprint comparison. The PLECFP, or protein-ligand extended connectivity fingerprint, is a method for hashing protein-ligand interactions for comparison. Here, I used it to compare structural interactions between the leads and acarbose during protein binding. Here are the results for each:Molecular decoysDecoys help us determine whether our docking results are false positives. By matching our lead molecules against decoy molecules with similar properties (e.g., molecular weight, number of rotatable bonds, log P), we can assess whether our leads have real potential.I used LUDe (LIDEB’s Useful Decoys) to generate decoys for all (putatively) biologically active ligands. LUDe provides fifty decoys per ligand, all of which were docked against the protein models using GNINA under the same parameters, with CNN-VS taken as well.Comparative overall performance (measured via CNN-VS) of leads vs. property-matched decoys.Alternatively:As you can see, nearly all decoy poses scored below our activity threshold. However, to be more robust, I did the following checks: Get a one-sided empirical p-value, robust z-score, and Benjamini-Hochberg adjusted q-value for each ligand vs. decoy set. Determine the top-1 enrichment for each ligand vs. decoy set (or, how often does the ligand outrank its decoys in CNN-VS). P-values & z-scoresFor each ligand, I compared CNN-VS scores with ~50 matched decoys (in some cases, RDKit was unable to generate conformers for the decoys) and computed a one-sided empirical p-value. I also determined how many robust standard deviations the ligand lay above the median and adjusted all p-values using the Benjamini-Hochberg false discovery rate. 2QMJ & 5NN8: For all ligands, all adjusted p-values < 0.05. 3BAJ : Out of 53 ligands, one p-value > 0.05. Some results for 3BAJ are shown below.Ligand dominance, or rank-based enrichmentRealistically, the ligand vs. decoy sets are too small for any real statistical robustness. A more useful heuristic is to compare the ligands’ ranks relative to their matched decoys when ordered by CNN-VS.For all three proteins, all leads ranked at or above the 98th percentile relative to their decoys. Actually, all except 2 leads for 3BAJ ranked first relative to their decoys. Some results for 3BAJ are shown below:You’ll notice very few decoys were generated for CNP0064234.2; it was not considered a candidate for 3BAJ.Redocking with variant seedsOne of the more important validation tests, this process is meant to tell us if the leads we determined were flukes or are consistently high-CNN-VS across replicates , an attribute we’ll refer to as stability from here on.Once again, this control helps us filter out false positives. If expanded, it can help us identify false negatives: ligands that didn’t pass the CNN-VS threshold in the original screen but would otherwise. I chose not to do that in this project due to time and computational cost constraints, but it would be a logical next step.For this control, I simply took all of our leads and docked them again with GNINA using 5 variant seeds. Below are the distributions of CNN-VS scores for the initial run compared with the redocked runs:As we can see, the CNN-VS for the redocked ligands exhibits a wider distribution (as expected — we haven’t artificially subset them to CNN-VS >= 6.30, and there are 25 times as many data points).Let’s take a closer look: 5.1. Jittered Box-Plot of CNN-VS Distribution by Identifier Across All SeedsWhat this shows:Each box shows the spread of CNN-VS scores for a given lead identifier across all seeds, including the original and redocking runs. Identifiers are ranked by median CNN-VS. This gives us a sense of consistency vs variance per ligand. Already, we can see several leads do not pass muster.Why it matters:These charts help us see which identifiers have the most consistent CNN-VS scores and whether their distributions exceed the biologically active threshold. 5.2. Spaghetti Plot of Best CNN-VS Score by Identifier Across All SeedsWhat this shows :This is a closer look than the previous boxplot and shows us seeds on the x-axis. Once again, it shows us the consistency and potency of each identifier: Which ligands are seed-stable vs. seed-sensitive? How often are trajectories exceeding 6.30? Are high medians due to lucky seeds or consistency? Note : It might make more sense to use a spaghetti plot for fewer samples that were actually labelled, but here it’s meant to provide a broader overview.Why it matters :These plots show whether CNN-VS scores are consistent across runs and how high or low those scores are. We’re looking for consistent scoring that’s also above the threshold. 5.3. Scatter Plot of Cross-Seed Median CNN-VS Score & Cross-Seed Interquartile RangeWhat this shows :The median CNN-VS across seeds provides a general idea of the CNN-VS for a particular ligand; the IQR indicates the stability of that CNN-VS. A lower IQR means a tighter distribution. Bottom-right quadrant: low variance, high CNN-VS (most promising candidates). Top-right quadrant: high variance, high CNN-VS (false positives). Top-left quadrant: high variance, low CNN-VS (discard). Bottom-left quadrant: low variance, low CNN-VS (true negatives) Are high medians due to lucky seeds or consistency? Why it matters :This is another way of visualizing consistency (low IQR) and quality (median CNN-VS). 5.4. Median Comparison Between Originals and Redock CNN-VS by IdentifierWhat this shows :One of the simpler visualizations; all we see here is the difference between the original run’s CNN-VS and the median redock runs’ CNN-VS. A small drop tells us our original run for that ligand was reliable. A big drop means it got lucky. This plot becomes more useful if we expand our initial redocking set to include those leads with CNN-VS scores below 6.30 — a jump would indicate a false negative.Why it matters :A companion to IQR vs. CNN-VS. A snapshot of pose stability.Taking all of this into account, my final filtration process was to subset to the leads that scored CNN-VS > 6.30 in 5 out of 6 replicates, then rank them by the median CNN-VS-to-interquartile range ratio. That meant the top 3 candidates for each protein were as follows:2QMJ : CNP0111740.3 CNP0475805.2 (Phaeospelide A) CNP0233949.3 (Convallataxol) 3BAJ : CNP0242138.5 CNP0146238.1 CNP0185842.19 5NN8 : CNP0533794.1 CNP0234543.1 (2-deoxyecdysterone 20,20-monoacetonide) CNP0111740.3 We’ll take a closer look at these soon — but first, let’s do one final validation.Structural interaction fingerprint comparisonThe goal of this step is more exploratory than confirmatory. My aim here was to determine whether the lead ligands interact with the target proteins similarly to experimental acarbose. We don’t exactly need to prioritize the ligands with similar binding profiles, as it could be expected that a more performant ligand would behave differently.We can compare ligand-protein structural interactions across a diverse set of ligand sizes using structural interaction fingerprints. These go one step beyond typical molecular fingerprints and encode molecular interactions between ligands and binding-pocket residues, hashing them to preset sizes to enable comparisons.It seems the best fingerprint so far isPLECFP, which combines the best features of all previous methods — atomic environment capture, expanded capture radius, etc. It’s shown promise in AI tasks, lead optimization, and scaffold hopping. That’s why I chose to use it for my initial pass:As you can see, our similarity scores are practically zero. (Full disclosure: this was a pretty disappointing chart to generate).These results are a little surprising — the initial list of 10,000 candidates was selected because they shared similarities with acarbose, so we’d expect at least some overlap.Moreover, when creating a residue contact map for each ligand against its target protein, the interacting residues seem quite similar:2QMJ :3BAJ :5NN8 :I’m still not sure why this might be, so please let me know if you have an explanation!Also, out of curiosity, I determined pairwise Tanimoto similarity across all leads2. There seems to be clustering among some 3BAJ leads, which may merit further investigation.Some of the lead ligands for 3BAJ, such as CNP0146238.29, appear to share binding behaviours with other leads, such as CNP0156494 — this may be something worth further investigation.Candidate overview & key takeawaysLet’s start by interpreting our shortlist from earlier:2QMJ CNP0111740.3 : An oleanane triterpenoid that shares parentage with Ilekudinoside I, and seems to appear in traditional Chinese medicine. CNP0475805.2 (Phaeospelide A): A polyene macrolide from Arthrinium phaeospermum , a hairy-caterpillar-associated fungus. Although the genes required to express this molecule appear silent in the fungus, they were heterologously expressed in bacteria. This molecule also exhibits binding affinity with 5NN8. CNP0233949.3 (Convallataxol): The best-annotated NP on this list, convallataxol, is a steroid glycoside found in the leaves and flowers of Antiaris toxicaria , a highly poisonous tree. 3BAJ CNP0242138.5 : Another cardenolide from the Nerium indicum and Nerium oleander species, flowering shrubs used in traditional medicine. CNP0146238.1 : An ecdysteroid that appears to be used in Ethiopian traditional medicine. CNP0185842.19 : A cardenolide that seems to be a stereo variant of beauwalloside, which is also found in N. oleander. 5NN8 CNP0533794.1: An open-chain polyketide — no other documentation found. CNP0234543.1 (2-deoxyecdysterone 20,20-monoacetonide): An ecdysteroid from the roots of the Silene brahuica plant, shown to have some anti-inflammatory properties. CNP0111740.3 (See earlier) Some honourable mentions: CNP0156494.1 : An ecdysteroid from Vitex polygama , which appears to be shidasterone, which seems to have enzyme-inhibitory effects. CNP0279421.3 : Torvoside K, a saponin from many species, including ones of the Solanum genus, which seems to have antifungal properties. If you’re in a lab, made it this far, and have access to such delightful flora as A. toxicaria or a library of traditional Ethiopian medicine, feel free to take this list of lead candidates and get to testing.Some of my key takeaways: The easiest part of virtual screening is the docking. The real work involves preparation, preprocessing, post-docking controls, and validation. Many natural products lack sufficient research, even though they present countless therapeutic opportunities. Admittedly, this is largely due to extraction and isolation challenges — but there’s an opportunity here nonetheless. Good research involves lots of small, smart decisions. Several times during this project (not documented here), I had to back up and redo an entire step after rushing through a small decision. Better planning and research ahead of time would have saved time and boosted my efficiency. Thank you for reading! Am I a genius? A modern Prometheus? Have I made grievous and unforgivable scientific errors? Please let me know if you have any feedback.1They actually specifically recommend against using GNINA for high-throughput screening applications. However, simple parallelization allows you to spawn multiple GNINA processes on a single GPU node, so you can take advantage of the program’s speed.22BAJ :3BAJ :5NN8:", "content_html": "

\"\"By Author — Not random images either, there’s a chance you find the compounds in all of these organisms.

\"Static

Table of Contents

  1. Exploratory Data Analysis for COCONUT — Profiling ~100K natural products to understand drug-likeness, physicochemical space, and screening viability.
  2. Chemical Similarity Search — Fingerprint-based filtering (ECFP, MACCS, pharmacophore, USRCAT) to reduce the database to high-similarity candidates.
  3. Protein Model Selection, Cleanup, and Preprocessing — Selecting acarbose-bound crystal structures and preparing them for docking.
  4. Protein–Ligand Docking & Assessment — Running GNINA-based virtual screening and scoring with CNN-VS.
  5. Lead Validation and Controls — Decoys, redocking replicates, statistical filtering, and stability analysis to remove false positives.
  6. Candidate Overview & Key Takeaways — Final shortlist by target protein and reflections on the workflow.

Flavonoids are a large, diverse group of secondary plant metabolites that play a role in pigmentation, UV protection, and insecticidal activity. Specifically, some anthocyanins (a subclass) can inhibit insect digestive enzymes, preventing nutrient uptake and development, serving as a defence mechanism against predators.

Obviously, there’s potential here for bug Ozempic (or insecticides, you pick). But research on Allium mongolicum flowers has also revealed that their flavonoids may inhibit human digestive enzymes, including our very own starch-metabolizing α-glucosidase. These compounds could be used to treat diabetes, as inhibiting starch breakdown reduces postprandial blood glucose levels (Li et al., 2025).

Initial docking analysis revealed that A. mongolicum-derived isoquercetin (already in clinical trials) can inhibit α-glucosidase (AGI) activity with an effect similar to that of acarbose , a common AGI derived from Actinoplanes bacteria. Chemical similarity search revealed that troxerutin (aka vitamin P4, a flavonoid you can buy over the counter) shares some substructural motifs with isoquercetin, and its docking analysis showed similar binding affinity.

\"\"Docking results from DiffDock-Pocket.

Note : troxerutin’s slightly higher affinity may be attributed to steric interactions introduced by the moieties found on the larger molecule.

I wanted to take this approach and extend it to a larger database of natural compounds to find other biochemicals that could hypothetically outperform acarbose — after all, the drug is projected to have a $220 million global market by 2035.

Here’s my overall process:

  1. Exploratory data analysis of COCONUT, a natural products database.

  2. Chemical similarity search of acarbose against COCONUT to find structurally similar NPs.

  3. Protein model selection, cleanup, and preprocessing.

  4. Protein-ligand docking and assessment , where I run a virtual screen against human α-glucosidases.

  5. Lead validation and controls to identify any false positives.

You can see all the code (in progress) here.

\"\" Workflow diagram

1. Exploratory data analysis for COCONUT

In this section, I’ll take a look at:

There are a little over 100,000 natural products in the COCONUT database. I started by doing some simple feature visualization:

\"\"

For a closer look at some derived ratios, I looked at the following:

\"\"

Research indicates that a TPSA/MW ratio >= 0.2 is ideal for good solubility; approximately half of the dataset meets this criterion (Whitty et al., 2018).

\"\"

This metric stems from two findings:

Here, I take the ratio of both in the dataset. Two populations emerge: many NPs with no saturated carbons and a smaller group with some balance.

Takeaways :

2. Chemical similarity search

Chemical similarity is generally determined by generating chemical fingerprints for a molecule, fixed-size identifiers that capture information about atomic structure and chemistry. Several fingerprinting protocols exist, each with its own strengths and weaknesses. Here are the ones I chose and why:

By generating fingerprints for acarbose, we can use Tanimoto similarity to assess the degree of agreement between our database compounds and the lead target. USR is the exception; we use inverse Manhattan distance instead.

To trim the dataset down, I used the following cutoffs for Tanimoto similarity:

This gave me a dataset of 10,341 high-similarity candidate molecules to explore. Ligands were generated using RDKit and converted to .pdbqt using OpenBabel.

3. Protein model selection, cleanup, and preprocessing

Acarbose is a bit of a rolling stone, it can’t be tied down — by which I mean it interacts with multiple proteins. To be thorough, I wanted to explore this virtual screening process across all of its protein targets at once. In hindsight, this made it much harder to keep track of (I’ll take it one target at a time from now on).

I made my selections based on protein targets listed in DrugBank and the resolution of the corresponding protein model. I also specifically selected PDB entries of the target protein _ in complex with acarbose_ so I’d a) have a defined binding pocket and b) have an experimental reference point for future validations. Finally, all proteins were put through PDB-redo before being converted to .pdbqt for docking:

  1. 2QMJ : The N-terminal subunit of human maltase-glucoamylase in complex with acarbose. As you can see from the PDB page, there’s room for growth in model quality, but PDB-redo led to some minor improvements. (Gene: MGAM).
  2. 3BAJ : Human pancreatic alpha-amylase in complex with nitrate and acarbose. PDB-redo led to more significant improvements here. (Gene: AMY2A).
  3. 5NN8 : Human lysosomal acid-alpha-glucosidase in complex with acarbose (Gene: GAA).

Below, acarbose is designated in light green with binding residues (within 4 Å of acarbose) in red. All other observed elements are ions and ligands, which are removed for docking.

\"\"Note: one of the highlighted light green ligands in 5NN8 is an acarbose-derived trisaccharide; the other binding pocket (containing true alpha-acarbose) is used for docking configuration — Images generated via The Protein Imager.

4. Protein-ligand docking & assessment

\"\"Figures generated using ChimeraX. Can you tell I tried (unsuccessfully) to create Goodsell-like graphics?

For protein-ligand docking, I usedGNINA, an open-source AutoDock Vina fork that uses CNNs to score ligand poses.

Since we know where acarbose is supposed to bind (the protein models include acarbose), we can guide the docking process by setting an autobox (basically a mini 3D search space) around its coordinates. In the diagrams above, I’ve indicated the autobox GNINA draws around the ligands (4 Å outward from the farthest corners of the molecule) in red.

While it may be the most computationally expensive step, this stage of the project was actually the easiest. I ran GNINA on an HPC cluster in parallel fashion to save time — although GNINA wasn’t designed specifically for high-throughput virtual screening1, this step was completed in approximately one day.

5. Lead validation and controls

In this section, I go over:

GNINA generates several metrics for binding poses , including CNN pose score (the AI’s assessment for score quality) and CNN binding affinity (the AI-predicted kcal/mol). Benchmarking studies with GNINA have indicated that the product of these two values, CNN_VS (virtual screening), can be used to identify potentially active ligands when it exceeds 6.30.

\"\"Horizontal lines represent the CNN-VS threshold of 6.30.

Taking the CNN_VS of all ligands docked against each protein, 71 ligands passed the threshold.

Okay, the headline promise-payoff is complete—but we’re not done yet. After collecting a set of leads, I needed to validate some of the findings. There were a few ways I went about doing this, with each control telling me something different:

Here are the results for each:

Molecular decoys

Decoys help us determine whether our docking results are false positives. By matching our lead molecules against decoy molecules with similar properties (e.g., molecular weight, number of rotatable bonds, log P), we can assess whether our leads have real potential.

I used LUDe (LIDEB’s Useful Decoys) to generate decoys for all (putatively) biologically active ligands. LUDe provides fifty decoys per ligand, all of which were docked against the protein models using GNINA under the same parameters, with CNN-VS taken as well.

\"\"Comparative overall performance (measured via CNN-VS) of leads vs. property-matched decoys.

Alternatively:

\"\"

As you can see, nearly all decoy poses scored below our activity threshold. However, to be more robust, I did the following checks:

P-values & z-scores

For each ligand, I compared CNN-VS scores with ~50 matched decoys (in some cases, RDKit was unable to generate conformers for the decoys) and computed a one-sided empirical p-value. I also determined how many robust standard deviations the ligand lay above the median and adjusted all p-values using the Benjamini-Hochberg false discovery rate.

Some results for 3BAJ are shown below.

\"\"

Ligand dominance, or rank-based enrichment

Realistically, the ligand vs. decoy sets are too small for any real statistical robustness. A more useful heuristic is to compare the ligands’ ranks relative to their matched decoys when ordered by CNN-VS.

For all three proteins, all leads ranked at or above the 98th percentile relative to their decoys. Actually, all except 2 leads for 3BAJ ranked first relative to their decoys. Some results for 3BAJ are shown below:

\"\"You’ll notice very few decoys were generated for CNP0064234.2; it was not considered a candidate for 3BAJ.

Redocking with variant seeds

One of the more important validation tests, this process is meant to tell us if the leads we determined were flukes or are consistently high-CNN-VS across replicates , an attribute we’ll refer to as stability from here on.

Once again, this control helps us filter out false positives. If expanded, it can help us identify false negatives: ligands that didn’t pass the CNN-VS threshold in the original screen but would otherwise. I chose not to do that in this project due to time and computational cost constraints, but it would be a logical next step.

For this control, I simply took all of our leads and docked them again with GNINA using 5 variant seeds. Below are the distributions of CNN-VS scores for the initial run compared with the redocked runs:

\"\"

As we can see, the CNN-VS for the redocked ligands exhibits a wider distribution (as expected — we haven’t artificially subset them to CNN-VS >= 6.30, and there are 25 times as many data points).

Let’s take a closer look:

\"\" 5.1. Jittered Box-Plot of CNN-VS Distribution by Identifier Across All Seeds

What this shows:

Each box shows the spread of CNN-VS scores for a given lead identifier across all seeds, including the original and redocking runs.

Already, we can see several leads do not pass muster.

Why it matters:

These charts help us see which identifiers have the most consistent CNN-VS scores and whether their distributions exceed the biologically active threshold.

\"\" 5.2. Spaghetti Plot of Best CNN-VS Score by Identifier Across All Seeds

What this shows :

This is a closer look than the previous boxplot and shows us seeds on the x-axis. Once again, it shows us the consistency and potency of each identifier:

Note : It might make more sense to use a spaghetti plot for fewer samples that were actually labelled, but here it’s meant to provide a broader overview.

Why it matters :

These plots show whether CNN-VS scores are consistent across runs and how high or low those scores are. We’re looking for consistent scoring that’s also above the threshold.

\"\" 5.3. Scatter Plot of Cross-Seed Median CNN-VS Score & Cross-Seed Interquartile Range

What this shows :

The median CNN-VS across seeds provides a general idea of the CNN-VS for a particular ligand; the IQR indicates the stability of that CNN-VS. A lower IQR means a tighter distribution.

Why it matters :

This is another way of visualizing consistency (low IQR) and quality (median CNN-VS).

\"\" 5.4. Median Comparison Between Originals and Redock CNN-VS by Identifier

What this shows :

One of the simpler visualizations; all we see here is the difference between the original run’s CNN-VS and the median redock runs’ CNN-VS.

This plot becomes more useful if we expand our initial redocking set to include those leads with CNN-VS scores below 6.30 — a jump would indicate a false negative.

Why it matters :

A companion to IQR vs. CNN-VS. A snapshot of pose stability.

Taking all of this into account, my final filtration process was to subset to the leads that scored CNN-VS > 6.30 in 5 out of 6 replicates, then rank them by the median CNN-VS-to-interquartile range ratio. That meant the top 3 candidates for each protein were as follows:

2QMJ :

3BAJ :

5NN8 :

We’ll take a closer look at these soon — but first, let’s do one final validation.

Structural interaction fingerprint comparison

The goal of this step is more exploratory than confirmatory. My aim here was to determine whether the lead ligands interact with the target proteins similarly to experimental acarbose. We don’t exactly need to prioritize the ligands with similar binding profiles, as it could be expected that a more performant ligand would behave differently.

We can compare ligand-protein structural interactions across a diverse set of ligand sizes using structural interaction fingerprints. These go one step beyond typical molecular fingerprints and encode molecular interactions between ligands and binding-pocket residues, hashing them to preset sizes to enable comparisons.

It seems the best fingerprint so far isPLECFP, which combines the best features of all previous methods — atomic environment capture, expanded capture radius, etc. It’s shown promise in AI tasks, lead optimization, and scaffold hopping. That’s why I chose to use it for my initial pass:

\"\"

As you can see, our similarity scores are practically zero. (Full disclosure: this was a pretty disappointing chart to generate).

These results are a little surprising — the initial list of 10,000 candidates was selected because they shared similarities with acarbose, so we’d expect at least some overlap.

Moreover, when creating a residue contact map for each ligand against its target protein, the interacting residues seem quite similar:

2QMJ :

\"\"

3BAJ :

\"\"

5NN8 :

\"\"

I’m still not sure why this might be, so please let me know if you have an explanation!

Also, out of curiosity, I determined pairwise Tanimoto similarity across all leads2. There seems to be clustering among some 3BAJ leads, which may merit further investigation.

Some of the lead ligands for 3BAJ, such as CNP0146238.29, appear to share binding behaviours with other leads, such as CNP0156494 — this may be something worth further investigation.

Candidate overview & key takeaways

Let’s start by interpreting our shortlist from earlier:

2QMJ

3BAJ

5NN8

Some honourable mentions:

If you’re in a lab, made it this far, and have access to such delightful flora as A. toxicaria or a library of traditional Ethiopian medicine, feel free to take this list of lead candidates and get to testing.

Some of my key takeaways:

Thank you for reading! Am I a genius? A modern Prometheus? Have I made grievous and unforgivable scientific errors? Please let me know if you have any feedback.

1

They actually specifically recommend against using GNINA for high-throughput screening applications. However, simple parallelization allows you to spawn multiple GNINA processes on a single GPU node, so you can take advantage of the program’s speed.

2

2BAJ :

\"\"

3BAJ :

\"\"

5NN8:

\"\"

", "url": "https://murto.co/acarbose-alternative-drug-discovery/", "image": "/img/seo/aadd.png", "tags": ["bioinformatics","drug-discovery","virtual-screening"], "date_published": "2026-02-11T00:00:00+00:00", "date_modified": "2026-02-11T00:00:00+00:00", "author": { "name": "Murto Hilali", "url": "https://murto.co/", "avatar": "https://murto.co/img/author/website-headshot.jpeg" } }, { "id": "https://murto.co/fusion-conference/", "title": "How to Win a Case Competition in 7 Hours", "summary": "Competing in and winning my first ever business innovation case competition.", "content_text": "By the way, a version of this article was posted to UWaterloo’s website, you can read it here.Fusion Conference 2025 was my first case competition (ever).Don’t make my mistake and wait until your final term to participate.You might just miss out on the chance to join an exceptional team, turn a blank slide deck into a healthtech pitch in under an hour, and propose an idea to a panel of industry experts — and win, while you’re at it.Proof? Here’s my experience at Fusion this year:The Fusion team did an excellent job handling arrival and registration. It was an incredibly diverse mix of attendees: accounting, science, and business students in equal measure. Soon, we heard from the day’s keynote speaker, UWaterloo alumnus Gavin Duggan — a senior engineer at Google who had worked at Alphabet’s moonshot factory, X.The keynote was a goldmine for anyone interested in healthtech innovation and business. And we’d need those insights soon, because the day’s topic was about to be revealed:Improving cancer services in Ontario.It’s a massive topic. Where do you even begin? With less than two hours on the clock, we had to: Understand our particular issue in Ontario Diagnose the problem with current solutions Propose a better one Wrap it up into a pitch and present to judgesPiece of cake, right?It’s 12:15. We landed on our idea twenty minutes ago, and we’ve got twenty minutes left to make it look presentable. Every new study we dig up forces us to pivot — strategy shifts, slides get rewritten, messaging gets reworked on the fly. We split up: three of us run the pitch, two grind through the deck. We’re pacing, practicing, stitching it all together as the timer ticks down.Easily the most stressful Saturday lunch of my life — and certainly one of the most fun.We submitted the deck with seconds to spare, survived a dramatic first pitch, and finally got to breathe — and eat the delicious conference lunch. The other teams came with wildly creative ideas: bioengineered sensors, AI carcinogen detection, all technology I was mentally bookmarking to research later.Then came the announcements for the six finalists.First team. Second. Third. Fourth. Fifth.Our row went silent.And then… our name.First, we celebrated! Then immediate panic. We had under an hour to expand the pitch, rebuild the slides, and prep for questions from a panel with decades of experience. With two hours of ‘experience’ under our belt, we got to work.We pitched last, after five other incredibly innovative ideas — drone delivery networks, AI for clinical trials, even whale-protein research. Anyone would’ve been nervous. But by then, we had figured out our workflow and learned to trust each other; we were running like a well-oiled machine.Then came the final announcement.Most innovative solution. $1,500 prize.The judges announced the winning idea — the same one we’d built together in an empty Hagey Hall classroom that morning.It was ours!Winning a prize at Fusion was obviously a highlight, but there were some takeaways from the day that I think any attendee can agree with: You can get a ridiculous amount of work done in a few hours when you’re thrown into a room full of smart, ambitious people and tasked with solving a big problem. The best ideas come from collaboration. Sharing half-formed thoughts and letting others build on them is what made our pitch work. Events like this are as much about the people as the problem. Some conversations lasted two minutes; others turned into exchanges I’m still following up on. If you want to meet more people in your program or industry, join a case competition like this one.My only regret? Not joining sooner.A huge thanks to the Fusion team for putting on a great conference, and of course, my incredible team members, Kiranjot, Roma, Inigo, and Yubin.", "content_html": "

\"HoldingBy the way, a version of this article was posted to UWaterloo’s website, you can read it here.

Fusion Conference 2025 was my first case competition (ever).

Don’t make my mistake and wait until your final term to participate.

You might just miss out on the chance to join an exceptional team, turn a blank slide deck into a healthtech pitch in under an hour, and propose an idea to a panel of industry experts — and win, while you’re at it.

Proof? Here’s my experience at Fusion this year:

\"\"

The Fusion team did an excellent job handling arrival and registration. It was an incredibly diverse mix of attendees: accounting, science, and business students in equal measure. Soon, we heard from the day’s keynote speaker, UWaterloo alumnus Gavin Duggan — a senior engineer at Google who had worked at Alphabet’s moonshot factory, X.

The keynote was a goldmine for anyone interested in healthtech innovation and business. And we’d need those insights soon, because the day’s topic was about to be revealed:

Improving cancer services in Ontario.

It’s a massive topic. Where do you even begin? With less than two hours on the clock, we had to:

  1. Understand our particular issue in Ontario
  2. Diagnose the problem with current solutions
  3. Propose a better one
  4. Wrap it up into a pitch and present to judges

Piece of cake, right?

\"\"It’s 12:15. We landed on our idea twenty minutes ago, and we’ve got twenty minutes left to make it look presentable. Every new study we dig up forces us to pivot — strategy shifts, slides get rewritten, messaging gets reworked on the fly. We split up: three of us run the pitch, two grind through the deck. We’re pacing, practicing, stitching it all together as the timer ticks down.

Easily the most stressful Saturday lunch of my life — and certainly one of the most fun.

\"\"We submitted the deck with seconds to spare, survived a dramatic first pitch, and finally got to breathe — and eat the delicious conference lunch. The other teams came with wildly creative ideas: bioengineered sensors, AI carcinogen detection, all technology I was mentally bookmarking to research later.

Then came the announcements for the six finalists.

First team. Second. Third. Fourth. Fifth.Our row went silent.

And then… our name.

First, we celebrated! Then immediate panic. We had under an hour to expand the pitch, rebuild the slides, and prep for questions from a panel with decades of experience. With two hours of ‘experience’ under our belt, we got to work.

\"\"

We pitched last, after five other incredibly innovative ideas — drone delivery networks, AI for clinical trials, even whale-protein research. Anyone would’ve been nervous. But by then, we had figured out our workflow and learned to trust each other; we were running like a well-oiled machine.

Then came the final announcement.

Most innovative solution. $1,500 prize.The judges announced the winning idea — the same one we’d built together in an empty Hagey Hall classroom that morning.

It was ours!

Winning a prize at Fusion was obviously a highlight, but there were some takeaways from the day that I think any attendee can agree with:

My only regret? Not joining sooner.

A huge thanks to the Fusion team for putting on a great conference, and of course, my incredible team members, Kiranjot, Roma, Inigo, and Yubin.

", "url": "https://murto.co/fusion-conference/", "image": "/img/seo/fusion-conference.png", "date_published": "2025-12-03T00:00:00+00:00", "date_modified": "2025-12-03T00:00:00+00:00", "author": { "name": "Murto Hilali", "url": "https://murto.co/", "avatar": "https://murto.co/img/author/website-headshot.jpeg" } }, { "id": "https://murto.co/uwaterloo-singapore-coop/", "title": "Interview with UWaterloo About International Co-op", "summary": "Got to do a profile for my University Science & Business department about doing computational chemistry research in Singapore.", "content_text": "After completing a computational chemistry co-op in Singapore, I was invited by the Science and Business department at the University of Waterloo to speak about my experience.You can check out the article here.", "content_html": "

\"\"After completing a computational chemistry co-op in Singapore, I was invited by the Science and Business department at the University of Waterloo to speak about my experience.

You can check out the article here.

", "url": "https://murto.co/uwaterloo-singapore-coop/", "image": "/img/seo/uwaterloo-singapore-coop.png", "date_published": "2025-02-10T00:00:00+00:00", "date_modified": "2025-02-10T00:00:00+00:00", "author": { "name": "Murto Hilali", "url": "https://murto.co/", "avatar": "https://murto.co/img/author/website-headshot.jpeg" } }, { "id": "https://murto.co/benchmarking-peppi-models/", "title": "PepBDB-ML Benchmark: ML vs CNN for Binding-Site Detection", "summary": "Assembling four different ML/DL models to compare performance on identifying peptide-protein binding sites.", "content_text": "Ali & Foreman, the Red Sox & the Yankees, Romeo & Juliet. (Wait a minute…)We love a good historical matchup, and in 2024, few are as important as the AI/LLM race. So, I wanted to throw my hat in the ring—and just so happened to have recently built a binding site detection dataset for proteins that I could use as a benchmark.The task? Given a protein structure, determine which parts of the sequence bind to other peptides.I assembled four different AI challengers to see which performed the best: RandomForest XGBoost IsolationForest Convolutional Neural Network (CNN)I’ll explain why this matters, why these aren’t language models — and, of course, the winners and losers.The BackgroundGoogle Trend data for ‘ChatGPT’Current hype is around language models — generative AI that can ‘read’ and write text. And the numbers speak for themselves: ChatGPT reached 100 million users in just two months. GPU manufacturer NVIDIA is now worth more than the GDP of Canada (my personal home, True North, strong and free). It’s estimated AI will add 21% to US GDP by 2030.LLMs are going to solve countless issues — but not all of them.Picking the right AI model for your task is crucial. And with LLMs, you might just be hiring an elephant to catch a mouse.Example: I do bioinformatics, so I work with biological data. Some of that data is tabular or graphic — meaning LLMs could be useful for analysis, but may require a disproportionate amount of computing power for training.As a classification/detection task, the compute-to-accuracy ratio for this problem makes a lot more sense if we use smaller models — like decision trees or CNNs.The DatasetThe PepBDB-ML dataset is derived from the PepBDB database, a curated collection of peptide-protein complexes from the RCSB PDB.PepBDB, ScreenshotThe dataset is inspired by a paper from Wardah et al., which collected residue-level data on protein sequence windows and turned them into images, using a CNN to classify them as binding or nonbinding.I generated my own version using structural features from the PepBDB:In the visualization above, columns are for amino acids, and the rows represent features like hydrophobicity or accessible surface area.On the way to creating images like these, however, we end up with tabular data. (It looks like this).This type of data is perfect for training a decision-tree style ML model, which explains my choice of contenders.Each row (and image) is labeled as either binding or nonbinding. Our challengers’ task is to correctly classify each one.Who are the challengers, you ask? Before we meet our fighters, let’s talk about the metrics we’re grading them on:AccuracyAccuracy is given by the formula:(true positives + true negatives) / total number of data points.In other words, how often does the model correctly classify the data?It’s crucial to remember that in the context of a peptide-protein binding site classification problem, accuracy isn’t the best metric.In a given dataset, it’s possible only 5% of the residues are binding — that means an algorithm predicting every single residue as non-binding will have a 95% accuracy rate. Highly accurate, minimally useful.PrecisionPrecision is given by the formula:true positives / (true positives + false positives)This measures how many of the positive predictions made by the model are actually correct.RecallRecall is given by the formula:true positives / (true positives + false negatives).This measures how many of the actual positive labels in the data are being correctly identified by the model. If it’s low, it means we’re missing several binding sites.F1 ScoreThe F1 score combines precision and recall into one metric, given by the formula:2 * ((precision * recall) / (precision + recall)).It is a better measure than accuracy, especially when dealing with imbalanced datasets.We’re going to keep a special eye out for recall: since binding sites are relatively rare compared to non-binding sites, we want to make sure our algorithm has a high sensitivity for detecting them. (Note: ‘sensitivity’ and ‘recall’ mean the same thing! Sensitivity is more commonly used in biological contexts).Housekeeping: for the four models below, I split PepBDB-ML into a training, testing, and validation set for the final metrics. The first three models were also hyperparameter-tuned, while I mostly relied on the established hyperparameters from the paper for the CNN.Now, let’s meet our challengers!The ChallengersChallenger #1: Rumblin’ RandomForestPress enter or click to view image in full size Type: Ensemble of decision trees. Strengths: Relatively simple, highly observable, flexible, and typically robust against overfitting. Weaknesses: It can be computationally expensive for many feature datasets, and it has limited accuracy relative to more sophisticated models.So how’d it do? Accuracy: 86.12% Precision: 49.36% Recall: 57.23% F1-Score: 53.01%Exceedingly average! Here’s a graphical overview:Confusion matrix for the RandomForest classifier’s performance | By AuthorAs you can see, the high accuracy is largely thanks to the high volume of true positives.Here’s the code for you to play around with yourself:## randomforest.pyimport pandas as pdimport numpy as npfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.metrics import accuracy_score, precision_score, recall_scorefrom sklearn.model_selection import RandomizedSearchCV, train_test_splitimport joblibfrom scipy.stats import randintimport matplotlib.pyplot as pltfrom sklearn.tree import export_graphvizfrom IPython.display import Imageimport graphviz# Random Forest Classification## 1. Open up the CSV file and split into features + targetsprint('Loading in data...')pepbdb = pd.read_csv('/path/to/peppi_data.csv')print('\\033[1mLoaded.\\033[0m')# One-hot encode the 'AA' columnpepbdb = pd.get_dummies(pepbdb, columns=['AA'])X = pepbdb.drop('Binding Indices', axis=1)y = pepbdb['Binding Indices']X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)## 2. Hyperparameter tuningprint('Starting hyperparameter tuning...')param_dist = { 'n_estimators': randint(100, 1000), 'max_depth': randint(3, 20), 'min_samples_split': randint(2, 20), 'min_samples_leaf': randint(1, 20), 'bootstrap': [True, False]}rf = RandomForestClassifier(n_jobs=-1, random_state=42, class_weight='balanced')rf_random = RandomizedSearchCV( estimator=rf, param_distributions=param_dist, n_iter=100, cv=3, verbose=2, random_state=42, n_jobs=-1, scoring='recall' )rf_random.fit(X_train, y_train)print('Best hyperparameters found:')print(rf_random.best_params_)best_rf = rf_random.best_estimator_print('Testing accuracy with the best model...')y_pred = best_rf.predict(X_test)accuracy = accuracy_score(y_test, y_pred)precision = precision_score(y_test, y_pred)recall = recall_score(y_test, y_pred)with open('/path/to/randomforest/summary_randomForest.txt', 'w') as f: f.write(f'Accuracy: {accuracy}\\n') f.write(f'Precision: {precision}\\n') f.write(f'Recall: {recall}')model_path = '/path/to/randomforest/random_forest_model_best.pkl'joblib.dump(best_rf, model_path)print('Visualizing...')for i in range(3): tree = best_rf.estimators_[i] dot_data = export_graphviz( tree, feature_names=X_train.columns, filled=True, max_depth=3, impurity=False, proportion=True ) graph = graphviz.Source(dot_data) file = f'/path/to/randomforest/tree_{i}' graph.render(file, format='png', cleanup=True)print(\"Graphs saved as PNG images.\")print('\\033[1mCompleted.\\033[0m')As you can see in the code above, one of the coolest parts of RandomForest is seeing the trees themselves:Decision tree from RandomForest ensemble | By AuthorChallenger #2: The Exceptional XGBoost Type: Gradient-boosted ensemble decision trees. Strengths: Gradient boosting involves the sequential building of trees, where each tree corrects on the previous one leading to better predictive performance. Also scales well to large datasets. Weaknesses: It might overfit to noise as it builds sequentially, and again, it is limited compared to even more sophisticated algorithms.XGBoost is an algorithm close to my heart, and it was one of my first forays into bio-ML. How’d it do? Accuracy: 76.54% Precision: 32.94% Recall: 69.02% F1-Score: 44.60%💔Heartbroken is an understatement. While our sensitivity (recall) has gone up, we’re still scoring pretty low on every other metric.Confusion matrix for the XGBoost classifier’s performance | By AuthorHere’s the code for your own exploration:## xgboostmodel.pyimport pandas as pdimport numpy as npimport joblibfrom sklearn.metrics import accuracy_score, confusion_matrix, precision_score, recall_score, f1_scorefrom sklearn.model_selection import RandomizedSearchCV, train_test_splitfrom scipy.stats import randint, uniformimport xgboost as xgbimport matplotlib.pyplot as pltimport seaborn as sns# Load dataprint('Loading in data...')pepbdb = pd.read_csv('/path/to/peppi_data.csv')print('\\033[1mLoaded.\\033[0m')# One-hot encode the 'AA' columnpepbdb = pd.get_dummies(pepbdb, columns=['AA'])X = pepbdb.drop('Binding Indices', axis=1)y = pepbdb['Binding Indices']# Split the data into training and test setsX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)# Calculate class weight rationum_pos = sum(y_train)num_neg = len(y_train) - num_posscale_pos_weight = num_neg / num_pos# Hyperparameter tuningprint('Starting hyperparameter tuning...')param_dist = { 'n_estimators': randint(100, 1000), 'max_depth': randint(3, 20), 'learning_rate': uniform(0.01, 0.2), 'subsample': uniform(0.6, 0.4), 'colsample_bytree': uniform(0.6, 0.4), 'gamma': uniform(0, 0.5)}xgb_model = xgb.XGBClassifier(objective='binary:logistic', n_jobs=-1, random_state=42, use_label_encoder=False, scale_pos_weight=scale_pos_weight)xgb_random = RandomizedSearchCV(estimator=xgb_model, param_distributions=param_dist, n_iter=100, cv=3, verbose=2, random_state=42, n_jobs=-1, scoring='recall')xgb_random.fit(X_train, y_train)print('Best hyperparameters found:')print(xgb_random.best_params_)best_xgb = xgb_random.best_estimator_print('Testing accuracy with the best model...')y_pred = best_xgb.predict(X_test)accuracy = accuracy_score(y_test, y_pred)precision = precision_score(y_test, y_pred)recall = recall_score(y_test, y_pred)fi_score = f1_score(y_test, y_pred)with open('/path/to/xgboostmodel/summary_xgboost.txt', 'w') as f: f.write(f'Accuracy: {accuracy}\\n') f.write(f'Precision: {precision}\\n') f.write(f'Recall: {recall}')# Save the model using XGBoost's save_model methodmodel_path = '/path/to/xgboostmodel/xgboost_model_best.json'best_xgb.save_model(model_path)print('\\033[1mCompleted.\\033[0m')Challenger #3: The Inimitatable IsolationForest Type: An anomaly detection algorithm also based on decision tree ensembles. Strengths: This architecture isolates anomalies, which could be useful since the rarity of binding sites may allow us to treat them as anomalies. Weaknesses: The algorithm assumes isolated anomalies — realistically, our anomalies (i.e. the binding sites) tend to be related.Despite the weaknesses, I thought it might be interesting to test how a model like this performs when we treat binding sites as anomalies. It’s also an unsupervised model, meaning it doesn’t need labeled features — useful for dirty data scenarios.By setting the contamination parameter (the predicted rate of anomalies) as our binding site class weight — how’d it do? Accuracy: 78.74% Precision: 22.21% Recall: 22.16% F1-Score: 22.18%Stunningly poor! This isn’t very surprising given what we know about the nature of binding sites and the purpose of this model.Confusion matrix for the IsolationForest classifier’s performance | By AuthorHere’s the code for you to experiment with. IsoForest doesn’t have binary outputs, so we map them to 1 and 0 here.(Note: this leads to an inflated custom F1 score that still needs debugging.)import pandas as pdimport numpy as npfrom sklearn.ensemble import IsolationForestfrom sklearn.model_selection import train_test_split, RandomizedSearchCVfrom sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, make_scorerimport joblibfrom scipy.stats import randintimport matplotlib.pyplot as pltprint('Loading in data...')pepbdb = pd.read_csv('/path/to/peppi_data.csv')print('\\033[1mLoaded.\\033[0m')# One-hot encode the 'AA' columnpepbdb = pd.get_dummies(pepbdb, columns=['AA'])X = pepbdb.drop('Binding Indices', axis=1)y = pepbdb['Binding Indices']# Calculate the proportion of binding residues (anomalies)contamination = y.mean()X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)print('Starting hyperparameter tuning...')param_dist = { 'n_estimators': randint(100, 1000), 'max_samples': randint(100, 500), 'max_features': randint(1, X.shape[1])}# Custom scoring functiondef custom_f1_score(y_true, y_pred): # Convert IsolationForest predictions to binary format y_pred_binary = np.where(y_pred == -1, 1, 0) return f1_score(y_true, y_pred_binary, average='weighted')if_base = IsolationForest(contamination=contamination, random_state=42)if_random = RandomizedSearchCV( estimator=if_base, param_distributions=param_dist, n_iter=100, cv=4, verbose=2, random_state=42, n_jobs=-1, scoring=make_scorer(custom_f1_score) # Use custom scoring function)if_random.fit(X_train, y_train)print('Best hyperparameters found:')print(if_random.best_params_)best_if = if_random.best_estimator_print('Testing classification with the best model...')y_pred = best_if.predict(X_test)# Convert predictions: -1 (anomaly/binding) to 1, 1 (normal/non-binding) to 0y_pred_binary = np.where(y_pred == -1, 1, 0)accuracy = accuracy_score(y_test, y_pred_binary)precision = precision_score(y_test, y_pred_binary)recall = recall_score(y_test, y_pred_binary)f1 = custom_f1_score(y_test, y_pred_binary)print(f'Accuracy: {accuracy}')print(f'Precision: {precision}')print(f'Recall: {recall}')print(f'F1 Score: {f1}')with open('/path/to/isolationforest/summary_isolationForest.txt', 'w') as f: f.write(f'Accuracy: {accuracy}\\n') f.write(f'Precision: {precision}\\n') f.write(f'Recall: {recall}\\n') f.write(f'F1 Score: {f1}\\n')model_path = '/path/to/isolationforest/isolation_forest_model_best.pkl'joblib.dump(best_if, model_path)# Plot anomaly scoresanomaly_scores = -best_if.score_samples(X_test)plt.figure(figsize=(10, 6))plt.hist([anomaly_scores[y_test == 0], anomaly_scores[y_test == 1]], label=['Non-binding', 'Binding'], bins=50, stacked=True)plt.title('Distribution of Anomaly Scores')plt.xlabel('Anomaly Score')plt.ylabel('Count')plt.legend()plt.savefig('/path/to/isolationforest/anomaly_scores_distribution.png')plt.close()print(\"Graphs saved as PNG images.\")print('\\033[1mCompleted.\\033[0m')Challenger #4: The Clamouring Convolutional Neural Network Type: A neural network that uses deep learning to classify images. Strengths: Incredibly useful for processing image data with a high degree of accuracy. This is the kind of AI used in many self-driving cars. Weaknesses: This is the most computationally expensive of our fighters and lacks interpretability — we can’t know why it makes the decisions it makes, unlike RandomForest or XGBoost.The genesis of this project was recreating Visual, a CNN built by Wardah et al. Building on its code, I modified the model to use PepBDB-ML images, a larger dataset with more features, including structural data.How did it do? Accuracy: 78.59% Precision: 78.48% (+ 10 pts)* Recall: 79.32% (+12 pts)* F1-Score: 50.26%Not too shabby! We also report an AUC (Area under ROC curve) of 0.86 (+ 5 pts)*, which tells us the model is fairly good at distinguishing between classes at various thresholds.*(compared to the original paper).Here’s what the confusion matrix looks like:Confusion matrix for Visual-PepBDB classifier’s performance.You’ll notice some totals don’t line up fully; this is because the image dataset is slightly smaller than the tabular dataset. The reasons are discussed in the article I wrote about it.Again, not great — but a step up from previous iterations.The code here is a bit more involved, but you can find it at Visual-PepBDB and play with it at Proteinloop.Press enter or click to view image in full sizeWhy did the CNN do better than everything else? Deep learning lets the model pick up on more nuanced relationships between data points. PepBDB-ML’s image dataset includes 7x41 windows of residues instead of the single entries we see in the table. This lets the CNN capture and understand more context around residues, providing more information to the model.We have our winner — Visual-PepBDB, or more broadly, the Convolutional Neural Network!Key Takeaways CNNs are by far the winner when it comes to detecting peptide-protein binding sites in PepBDB-ML — at least compared to RandomForest, XGBoost, and IsoForest. LLMs are tempting, trendy, and incredibly useful — but they may not be the right tools for your specific task. Traditional ML models can still be highly accurate, inexpensive, and more suited to your problem.If you’re in biology — I highly recommend exploring ML methods to see how they can augment your research. They don’t need to be cutting-edge, as code from 5+ years could be a game changer for your work. AI is much more approachable than you think!If you’re in ML/DL/AI—consider your entire toolbox, and apply it to biological problems! It’s filled with opportunities to apply classics and SOTA models.If you made it all the way here, thanks for reading! Let me know what you think.", "content_html": "

\"altAli & Foreman, the Red Sox & the Yankees, Romeo & Juliet. (Wait a minute…)

We love a good historical matchup, and in 2024, few are as important as the AI/LLM race. So, I wanted to throw my hat in the ring—and just so happened to have recently built a binding site detection dataset for proteins that I could use as a benchmark.

The task? Given a protein structure, determine which parts of the sequence bind to other peptides.

I assembled four different AI challengers to see which performed the best:

I’ll explain why this matters, why these aren’t language models — and, of course, the winners and losers.

The Background

\"GoogleGoogle Trend data for ‘ChatGPT’

Current hype is around language models — generative AI that can ‘read’ and write text. And the numbers speak for themselves:

LLMs are going to solve countless issues — but not all of them.

Picking the right AI model for your task is crucial. And with LLMs, you might just be hiring an elephant to catch a mouse.

Example: I do bioinformatics, so I work with biological data. Some of that data is tabular or graphic — meaning LLMs could be useful for analysis, but may require a disproportionate amount of computing power for training.

As a classification/detection task, the compute-to-accuracy ratio for this problem makes a lot more sense if we use smaller models — like decision trees or CNNs.

The Dataset

The PepBDB-ML dataset is derived from the PepBDB database, a curated collection of peptide-protein complexes from the RCSB PDB.

\"PepBDBPepBDB, Screenshot

The dataset is inspired by a paper from Wardah et al., which collected residue-level data on protein sequence windows and turned them into images, using a CNN to classify them as binding or nonbinding.

I generated my own version using structural features from the PepBDB:

\"Wardah

In the visualization above, columns are for amino acids, and the rows represent features like hydrophobicity or accessible surface area.

On the way to creating images like these, however, we end up with tabular data. (It looks like this).

This type of data is perfect for training a decision-tree style ML model, which explains my choice of contenders.

Each row (and image) is labeled as either binding or nonbinding. Our challengers’ task is to correctly classify each one.

Who are the challengers, you ask? Before we meet our fighters, let’s talk about the metrics we’re grading them on:

Accuracy

Accuracy is given by the formula:

(true positives + true negatives) / total number of data points.

In other words, how often does the model correctly classify the data?

It’s crucial to remember that in the context of a peptide-protein binding site classification problem, accuracy isn’t the best metric.

In a given dataset, it’s possible only 5% of the residues are binding — that means an algorithm predicting every single residue as non-binding will have a 95% accuracy rate. Highly accurate, minimally useful.

Precision

Precision is given by the formula:

true positives / (true positives + false positives)

This measures how many of the positive predictions made by the model are actually correct.

Recall

Recall is given by the formula:

true positives / (true positives + false negatives).

This measures how many of the actual positive labels in the data are being correctly identified by the model. If it’s low, it means we’re missing several binding sites.

F1 Score

The F1 score combines precision and recall into one metric, given by the formula:

2 * ((precision * recall) / (precision + recall)).

It is a better measure than accuracy, especially when dealing with imbalanced datasets.

We’re going to keep a special eye out for recall: since binding sites are relatively rare compared to non-binding sites, we want to make sure our algorithm has a high sensitivity for detecting them.

(Note: ‘sensitivity’ and ‘recall’ mean the same thing! Sensitivity is more commonly used in biological contexts).

Housekeeping: for the four models below, I split PepBDB-ML into a training, testing, and validation set for the final metrics. The first three models were also hyperparameter-tuned, while I mostly relied on the established hyperparameters from the paper for the CNN.

Now, let’s meet our challengers!

The Challengers

Challenger #1: Rumblin’ RandomForest

Press enter or click to view image in full size

\"\"

So how’d it do?

\"\"Confusion matrix for the RandomForest classifier’s performance | By Author

As you can see, the high accuracy is largely thanks to the high volume of true positives.

Here’s the code for you to play around with yourself:

## randomforest.pyimport pandas as pdimport numpy as npfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.metrics import accuracy_score, precision_score, recall_scorefrom sklearn.model_selection import RandomizedSearchCV, train_test_splitimport joblibfrom scipy.stats import randintimport matplotlib.pyplot as pltfrom sklearn.tree import export_graphvizfrom IPython.display import Imageimport graphviz# Random Forest Classification## 1. Open up the CSV file and split into features + targetsprint('Loading in data...')pepbdb = pd.read_csv('/path/to/peppi_data.csv')print('\\033[1mLoaded.\\033[0m')# One-hot encode the 'AA' columnpepbdb = pd.get_dummies(pepbdb, columns=['AA'])X = pepbdb.drop('Binding Indices', axis=1)y = pepbdb['Binding Indices']X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)## 2. Hyperparameter tuningprint('Starting hyperparameter tuning...')param_dist = {    'n_estimators': randint(100, 1000),    'max_depth': randint(3, 20),    'min_samples_split': randint(2, 20),    'min_samples_leaf': randint(1, 20),    'bootstrap': [True, False]}rf = RandomForestClassifier(n_jobs=-1, random_state=42, class_weight='balanced')rf_random = RandomizedSearchCV(    estimator=rf,     param_distributions=param_dist,     n_iter=100,     cv=3,     verbose=2,     random_state=42,     n_jobs=-1,    scoring='recall'    )rf_random.fit(X_train, y_train)print('Best hyperparameters found:')print(rf_random.best_params_)best_rf = rf_random.best_estimator_print('Testing accuracy with the best model...')y_pred = best_rf.predict(X_test)accuracy = accuracy_score(y_test, y_pred)precision = precision_score(y_test, y_pred)recall = recall_score(y_test, y_pred)with open('/path/to/randomforest/summary_randomForest.txt', 'w') as f:    f.write(f'Accuracy: {accuracy}\\n')    f.write(f'Precision: {precision}\\n')    f.write(f'Recall: {recall}')model_path = '/path/to/randomforest/random_forest_model_best.pkl'joblib.dump(best_rf, model_path)print('Visualizing...')for i in range(3):    tree = best_rf.estimators_[i]    dot_data = export_graphviz(        tree,        feature_names=X_train.columns,        filled=True,        max_depth=3,        impurity=False,        proportion=True    )    graph = graphviz.Source(dot_data)        file = f'/path/to/randomforest/tree_{i}'    graph.render(file, format='png', cleanup=True)print(\"Graphs saved as PNG images.\")print('\\033[1mCompleted.\\033[0m')

As you can see in the code above, one of the coolest parts of RandomForest is seeing the trees themselves:

\"\"Decision tree from RandomForest ensemble | By Author

Challenger #2: The Exceptional XGBoost

\"\"

XGBoost is an algorithm close to my heart, and it was one of my first forays into bio-ML. How’d it do?

Heartbroken is an understatement. While our sensitivity (recall) has gone up, we’re still scoring pretty low on every other metric.

\"\"Confusion matrix for the XGBoost classifier’s performance | By Author

Here’s the code for your own exploration:

## xgboostmodel.pyimport pandas as pdimport numpy as npimport joblibfrom sklearn.metrics import accuracy_score, confusion_matrix, precision_score, recall_score, f1_scorefrom sklearn.model_selection import RandomizedSearchCV, train_test_splitfrom scipy.stats import randint, uniformimport xgboost as xgbimport matplotlib.pyplot as pltimport seaborn as sns# Load dataprint('Loading in data...')pepbdb = pd.read_csv('/path/to/peppi_data.csv')print('\\033[1mLoaded.\\033[0m')# One-hot encode the 'AA' columnpepbdb = pd.get_dummies(pepbdb, columns=['AA'])X = pepbdb.drop('Binding Indices', axis=1)y = pepbdb['Binding Indices']# Split the data into training and test setsX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)# Calculate class weight rationum_pos = sum(y_train)num_neg = len(y_train) - num_posscale_pos_weight = num_neg / num_pos# Hyperparameter tuningprint('Starting hyperparameter tuning...')param_dist = {    'n_estimators': randint(100, 1000),    'max_depth': randint(3, 20),    'learning_rate': uniform(0.01, 0.2),    'subsample': uniform(0.6, 0.4),    'colsample_bytree': uniform(0.6, 0.4),    'gamma': uniform(0, 0.5)}xgb_model = xgb.XGBClassifier(objective='binary:logistic', n_jobs=-1, random_state=42, use_label_encoder=False, scale_pos_weight=scale_pos_weight)xgb_random = RandomizedSearchCV(estimator=xgb_model, param_distributions=param_dist, n_iter=100, cv=3, verbose=2, random_state=42, n_jobs=-1, scoring='recall')xgb_random.fit(X_train, y_train)print('Best hyperparameters found:')print(xgb_random.best_params_)best_xgb = xgb_random.best_estimator_print('Testing accuracy with the best model...')y_pred = best_xgb.predict(X_test)accuracy = accuracy_score(y_test, y_pred)precision = precision_score(y_test, y_pred)recall = recall_score(y_test, y_pred)fi_score = f1_score(y_test, y_pred)with open('/path/to/xgboostmodel/summary_xgboost.txt', 'w') as f:    f.write(f'Accuracy: {accuracy}\\n')    f.write(f'Precision: {precision}\\n')    f.write(f'Recall: {recall}')# Save the model using XGBoost's save_model methodmodel_path = '/path/to/xgboostmodel/xgboost_model_best.json'best_xgb.save_model(model_path)print('\\033[1mCompleted.\\033[0m')

Challenger #3: The Inimitatable IsolationForest

\"\"

Despite the weaknesses, I thought it might be interesting to test how a model like this performs when we treat binding sites as anomalies. It’s also an unsupervised model, meaning it doesn’t need labeled features — useful for dirty data scenarios.

By setting the contamination parameter (the predicted rate of anomalies) as our binding site class weight — how’d it do?

\"\"Confusion matrix for the IsolationForest classifier’s performance | By Author

Here’s the code for you to experiment with. IsoForest doesn’t have binary outputs, so we map them to 1 and 0 here.

(Note: this leads to an inflated custom F1 score that still needs debugging.)

import pandas as pdimport numpy as npfrom sklearn.ensemble import IsolationForestfrom sklearn.model_selection import train_test_split, RandomizedSearchCVfrom sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, make_scorerimport joblibfrom scipy.stats import randintimport matplotlib.pyplot as pltprint('Loading in data...')pepbdb = pd.read_csv('/path/to/peppi_data.csv')print('\\033[1mLoaded.\\033[0m')# One-hot encode the 'AA' columnpepbdb = pd.get_dummies(pepbdb, columns=['AA'])X = pepbdb.drop('Binding Indices', axis=1)y = pepbdb['Binding Indices']# Calculate the proportion of binding residues (anomalies)contamination = y.mean()X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)print('Starting hyperparameter tuning...')param_dist = {    'n_estimators': randint(100, 1000),    'max_samples': randint(100, 500),    'max_features': randint(1, X.shape[1])}# Custom scoring functiondef custom_f1_score(y_true, y_pred):    # Convert IsolationForest predictions to binary format    y_pred_binary = np.where(y_pred == -1, 1, 0)    return f1_score(y_true, y_pred_binary, average='weighted')if_base = IsolationForest(contamination=contamination, random_state=42)if_random = RandomizedSearchCV(    estimator=if_base,     param_distributions=param_dist,     n_iter=100,     cv=4,     verbose=2,     random_state=42,     n_jobs=-1,    scoring=make_scorer(custom_f1_score)  # Use custom scoring function)if_random.fit(X_train, y_train)print('Best hyperparameters found:')print(if_random.best_params_)best_if = if_random.best_estimator_print('Testing classification with the best model...')y_pred = best_if.predict(X_test)# Convert predictions: -1 (anomaly/binding) to 1, 1 (normal/non-binding) to 0y_pred_binary = np.where(y_pred == -1, 1, 0)accuracy = accuracy_score(y_test, y_pred_binary)precision = precision_score(y_test, y_pred_binary)recall = recall_score(y_test, y_pred_binary)f1 = custom_f1_score(y_test, y_pred_binary)print(f'Accuracy: {accuracy}')print(f'Precision: {precision}')print(f'Recall: {recall}')print(f'F1 Score: {f1}')with open('/path/to/isolationforest/summary_isolationForest.txt', 'w') as f:    f.write(f'Accuracy: {accuracy}\\n')    f.write(f'Precision: {precision}\\n')    f.write(f'Recall: {recall}\\n')    f.write(f'F1 Score: {f1}\\n')model_path = '/path/to/isolationforest/isolation_forest_model_best.pkl'joblib.dump(best_if, model_path)# Plot anomaly scoresanomaly_scores = -best_if.score_samples(X_test)plt.figure(figsize=(10, 6))plt.hist([anomaly_scores[y_test == 0], anomaly_scores[y_test == 1]],          label=['Non-binding', 'Binding'], bins=50, stacked=True)plt.title('Distribution of Anomaly Scores')plt.xlabel('Anomaly Score')plt.ylabel('Count')plt.legend()plt.savefig('/path/to/isolationforest/anomaly_scores_distribution.png')plt.close()print(\"Graphs saved as PNG images.\")print('\\033[1mCompleted.\\033[0m')

Challenger #4: The Clamouring Convolutional Neural Network

\"\"

The genesis of this project was recreating Visual, a CNN built by Wardah et al. Building on its code, I modified the model to use PepBDB-ML images, a larger dataset with more features, including structural data.

How did it do?

Not too shabby! We also report an AUC (Area under ROC curve) of 0.86 (+ 5 pts)*, which tells us the model is fairly good at distinguishing between classes at various thresholds.

*(compared to the original paper).

Here’s what the confusion matrix looks like:

\"\"Confusion matrix for Visual-PepBDB classifier’s performance.

You’ll notice some totals don’t line up fully; this is because the image dataset is slightly smaller than the tabular dataset. The reasons are discussed in the article I wrote about it.

Again, not great — but a step up from previous iterations.

The code here is a bit more involved, but you can find it at Visual-PepBDB and play with it at Proteinloop.

Press enter or click to view image in full size

\"\"

Why did the CNN do better than everything else?

We have our winner — Visual-PepBDB, or more broadly, the Convolutional Neural Network!

\"\"

Key Takeaways

If you’re in biology — I highly recommend exploring ML methods to see how they can augment your research. They don’t need to be cutting-edge, as code from 5+ years could be a game changer for your work. AI is much more approachable than you think!

If you’re in ML/DL/AI—consider your entire toolbox, and apply it to biological problems! It’s filled with opportunities to apply classics and SOTA models.

If you made it all the way here, thanks for reading! Let me know what you think.

", "url": "https://murto.co/benchmarking-peppi-models/", "image": "/img/seo/benchmarking-peppi-models.png", "tags": ["ai","bioinformatics"], "date_published": "2024-08-31T00:00:00+00:00", "date_modified": "2024-08-31T00:00:00+00:00", "author": { "name": "Murto Hilali", "url": "https://murto.co/", "avatar": "https://murto.co/img/author/website-headshot.jpeg" } }, { "id": "https://murto.co/pepbdb-ml/", "title": "How I Built A Custom Protein-Binding Dataset for Machine Learning", "summary": "A walkthrough on building an ML-friendly dataset with code, data visualizations, and cited sources.", "content_text": "Table of Contents Table of Contents Background: CNNs can predict binding sites The Rationale: But wait — does the data really matter that much? 1. Availability 2. Quality 3. Interpretability The Process: Building the PepBDB-ML dataset 1. Initial filtering 2. Sequence extraction using BioPython 3. Binding sequences determination with PRODIGY 4. Adding sequence level information with AAindex and PSI-BLAST AAindex PSI-BLAST 5. Structural information using BioPython Secondary structure codes, half-sphere exposure, and from DSSP Optional: Generating an image dataset The Results: Visualizing the PepBDB-ML database Amino acid distributions Correlation mapping Feature distributions Statistical tests The Discussion: Key takeaways and learnings90% of the time I tell people I work with proteins, they assume I’m trying to get jacked.And okay, fine, they’re right.But really, I’m talking about biological proteins— incredible molecular machines that make it possible for your body to function.Proteins are involved in vital metabolic processes — including getting that summer body — and peptide-protein interactions (PepPIs) are among the most important.(Peptide = short protein segments).Recently, I built a dataset I could use to study PepPIs. And by the end of this article, you will have too.Here’s a quick peek at a few things I discovered in what I’m calling the PepBDB-ML database: Protein residue pseudo angles take on a bimodal distribution. Substitution scores for residues are normally distributed (but there is one exception). Amino acids that are typically found in beta sheets might be more open to swapping with isoleucine than other residues. But I’m getting ahead of myself —Let’s start with some background.Background: CNNs can predict binding sitesRecently, I read a paper that used CNNs to predict peptide-protein binding sites from a sequence. They took a context window from the sequence (7 amino acids, AKA residues), enriched it with data on hydrophobicity, exposure, relative angles, etc., and transformed it into an image. Repeating this across a body of residues created a massive dataset of training images. Analogy: Imagine you had a Lego Deathstar, and you took a picture of each brick — and these pictures were force-imbued, so you could see things like the age and weight of each brick. Label each image with data about where it goes in the Deathstar (e.g., interior vs exterior), and you’d learn how to classify new pieces. Protein = Deathstar, Lego brick = amino acid Feeding this into a CNN yielded a model with binding site detection rates twice as high as previously published methods.I thought this approach was fascinating, but I saw an opportunity for improvement: residue secondary structure tags, angles, and accessible surface area were all predicted using SPIDER2 — but what if I just got the actual data from experimental structures instead?That’s precisely what I did using the PepBDB, a structural database of 3D peptide-receptor complexes (all structures are curated from the RCSB PDB, which provides data under license CC0).By using all the information applied in Visual, along with some structure-specific features, I could: Expand the dataset Potentially improve model performance Save the world (Two truths, one lie).So that is precisely what I set out to do with PepBDB-MLThe Rationale: But wait — does the data really matter that much?Well, there’s one key reason I think focusing on data is one of the highest-leverage activities in AI: data is our greatest constraint.Sure, compute is critical — but the data hydra has three heads:1. AvailabilityHeard about the recent content licensing deal between OpenAI and News Corp, Vox Media, and The Atlantic? The agreement with News Corp alone is believed to be worth over $250 million.While a lot of AI training data is either unprotected or open-source, the good stuff is often unavailable, expensive, or hard to find. Which leads to the next head…2. QualityLet’s use an example.If LLMs right now are really good at generating text, why not use them to create more training data? Unfortunately, making AI eat its own tail causes something called Model Autophagy Disorder, or MAD — and yes, they came up with the acronym before deciding what it stood for.Bottom line? Synthetic data can only take us so far. In the world of biology, that means we need reliable, experimental data (like the structures from PepBDB).3. InterpretabilityIn biology specifically, human-readable data is often the most helpful. It allows us to unpack machine learning models and see things like: Critical features. Relative importance. The future (I’m only half-joking). For example, when it comes to peptide-protein binding, what’s more impactful — hydrophobicity or half-sphere exposure? Without labels, we can’t answer that question.Documented data means that models and humans can both learn more about biology.Key Points: We can always fine-tune ML/DL models, but we’re bottlenecked by the quality of our data. Availability, quality, and interpretability are crucial aspects of biological datasets. Now that you’re convinced we’re not wasting time, let’s get into the meat of this project: building the dataset.The Process: Building the PepBDB-ML datasetBefore we begin, let’s decide what we want our dataset to look like. A tabular version, where each row represents a residue from a protein complex and each column is a feature (like residue hydrophobicity, polarizability, secondary structure within the protein). These features will be the ones used in Visual, along with some extras. An image version, where each image is essentially 7 adjacent rows from the tabular dataset, representing a context window for the central amino acids. With this plan in mind, we can focus our efforts on the tabular dataset. Since the PepBDB database contains structures, we can use structural and sequence information. Here’s how we’ll approach this: Initial filtering Sequence extraction + filtering Binding residue determination with Prodigy Sequence level information (AAindex, PSI-BLAST) Structural information from BioPython (DSSP, HSE, Pseudo-angles) There’s going to be a lot of code coming up, so buckle up.Before we begin, we’ll start with our imports: # gendata.py import pandas as pd import numpy as np import argparse import os import pickle from aaindex import * from helpers import * from paths import *What are those ones at the bottom? Don’t worry — we’ll get to those soon.(Heads up — since we’re going to be jumping between a few scripts, I’ll include the filename at the top of each code block).1. Initial filteringThe PepBDB offers the peptidelist.txt file as part of its download. This file provides information about each peptide-protein pair in the repository. # gendata.py ## Load the peptidelist.txt file peptide_list = pd.read_csv(peptide_list_txt, sep='\\\\\\\\s+', header=None) headers = ['PDB ID', 'Peptide Chain ID', 'Peptide Length', 'Number of Atoms in Peptide', 'Protein Chain ID', 'Number of Atoms in Protein', 'Number of Atom Contacts', 'unknown1', 'unknown2', 'Resolution', 'Molecular Type'] peptide_list.columns = headers # 'pepbdb' below is the path to the PepBDB directory, imported from `paths` peptide_list['Peptide Path'] = pepbdb + peptide_list['PDB ID'] + '_' + peptide_list['Peptide Chain ID'] + '/peptide.pdb' peptide_list['Protein Path'] = pepbdb + peptide_list['PDB ID'] + '_' + peptide_list['Peptide Chain ID'] + '/receptor.pdb' peptide_list.reset_index(drop=True) ## Filter out nucleic acids, low-resolution, and small peptides peptide_list = peptide_list[peptide_list['Molecular Type'] != 'prot-nuc'] peptide_list = peptide_list[peptide_list['Resolution'] < 2.5] peptide_list = peptide_list[peptide_list['Peptide Length'] >= 10] print('\\\\033[1mInitial filtering done.\\\\033[0m')We’ll do some very simple filtering to begin: Drop all prot-nuc models. These are models that contain a nucleic acid. We’re restricting this dataset to protein-only for the sake of simplicity. Drop models with resolution higher than 2.5 Å. We want our input data to be high-quality, so let’s drop any low-res models. Drop models with extra short peptides. This is mainly for the sake of computational efficiency. We’ll drop any peptides shorter than ten amino acids. In the next step, we’ll use BioPython to extract the sequences from each of our remaining models. Sequence extraction can take some time, which is why I’ve chosen to do it after some initial cleanup.2. Sequence extraction using BioPythonPepBDB provides 3D models but not direct sequences. Fortunately, these are relatively simple to extract using BioPython.To keep the code clean, I’ll create a helper function in a separate file called helpers.py: # helpers.py def extract_sequence(pdb_filename: str) -> str: ''' Helper function that extracts the sequence from a PDB file. ''' parser = PDBParser() structure = parser.get_structure('X', pdb_filename) sequence = '' contains_unk = False for model in structure: for chain in model: residues = chain.get_residues() for residue in residues: if residue.get_resname() == 'HOH': # ignoring water continue if residue.get_resname() == 'UNK': sequence += 'X' # add X for each 'UNK' residue else: sequence += seq1(residue.get_resname()) # seq1 converts 3-letter code to 1-letter code return sequenceThen, I’ll import it into our data generation script and apply it to our dataset: # gendata.py peptide_list['Peptide Sequence'] = peptide_list['Peptide Path'].apply(extract_sequence) peptide_list['Protein Sequence'] = peptide_list['Protein Path'].apply(extract_sequence) print('\\\\\\\\033[1mSequences extracted.\\\\\\\\033[0m')Now that we have sequences for each PDB structure, let’s remove the problematic ones (sequences with non-standard amino acids): # gendata.py peptide_list = peptide_list[~peptide_list['Peptide Sequence'].str.contains('[UOBZJX\\\\\\\\\\\\\\\\*]', regex=True)] peptide_list = peptide_list[~peptide_list['Protein Sequence'].str.contains('[UOBZJX\\\\\\\\\\\\\\\\*]', regex=True)] peptide_list = peptide_list.drop_duplicates(subset=['Peptide Sequence', 'Protein Sequence'])3. Binding sequences determination with PRODIGYPRODIGY (PROtein binDIng enerGY prediction) is a tool for predicting binding affinities for protein structures from 3D data.Part of that involves identifying residues in contact with each other — i.e., involved in binding.I used PRODIGY on every PDB in the PepBDB leftover from the last preprocessing step. The program outputs a .icfile — a simple text file that lists the residues involved in binding.Here’s an example for calcium-bound cyclin (1A03): GLY 78 A GLN 71 B ILE 13 A TYR 84 B LYS 40 A MET 1 B GLU 86 A HIS 27 B LEU 88 A LEU 12 B ALA 8 A ILE 15 B MET 1 A LYS 40 B ALA 87 A ILE 13 B PHE 70 A LEU 77 B GLN 71 A MET 82 B ALA 8 A ALA 8 B ILE 74 A LEU 77 B LYS 89 A LYS 26 BProcessing this data gives us two lists of indices, each referring to a binding residue.The peptides and receptors in each row are available as distinct PDBs, i.e. their models contain one chain each. For our purposes, PRODIGY needs a complex (2 chains), so we’ll make a temporary file and process the output: # helpers.py def label_residues(peptide_path: str, protein_path: str) -> List: # creating a temporary file with the peptide and protein with tempfile.NamedTemporaryFile(suffix='.pdb', delete=False) as temp_file: output_path = temp_file.name subprocess.run(['cat', peptide_path, protein_path], stdout=temp_file) # ensuring the temp file is properly closed before using it in subprocess temp_file.close() # obtaining binding residues using PRODIGY try: subprocess.run(['prodigy', '-q', '--contact_list', output_path], check=True) except subprocess.CalledProcessError as e: print(f\"An error occurred while processing the files: {peptide_path} & {protein_path}\") print(f\"Error: {e}\") # the .ic file will have the same root name as the input file ic_file_path = output_path.replace('.pdb', '.ic') # check if the .ic file exists before reading if not os.path.exists(ic_file_path): raise FileNotFoundError(f\"{ic_file_path} not found after running PRODIGY.\") # are you actually reading these comments? kudos! if yes — reply with a 😼 contacts = pd.read_csv(ic_file_path, sep='\\s+', header=None) contacts.columns = ['peptide_residue', 'peptide_index', 'peptide_chain', 'protein_residue', 'protein_index', 'protein_chain'] peptide_binding_residues = sorted(list(contacts['peptide_index'].unique())) protein_binding_residues = sorted(list(contacts['protein_index'].unique())) os.remove(output_path) return peptide_binding_residues, protein_binding_residuesNow, let’s apply it to our dataset: # gendata.py print('Running PRODIGY to determine binding contacts...') ## determine binding residues from peptide-protein complexes peptide_list['Peptide Binding Indices'] = np.nan peptide_list['Protein Binding Indices'] = np.nan for index, row in peptide_list.iterrows(): peptide_path = row['Peptide Path'] protein_path = row['Protein Path'] peptide_binding_positions, protein_binding_positions = label_residues(peptide_path, protein_path) peptide_list.at[index, 'Peptide Binding Indices'] = str(peptide_binding_positions) peptide_list.at[index, 'Protein Binding Indices'] = str(protein_binding_positions) print('\\\\\\\\033[1mBinding indices identified.\\\\\\\\033[0m') The Biology: You might be wondering — why build this dataset if we can use PRODIGY to get binding sites? Well, PRODIGY generally needs two chains in complex. Our dataset is aimed at helping ML/DL models predict binding sites with just one model chain.4. Adding sequence level information with AAindex and PSI-BLASTAAindexWe’ve now got a list of protein sequences. We can enrich our dataset by adding publically available, residue-specific information from the Amino Acid Index Database — a cornucopia of known, reference amino acid features.For example, here’s hydrophobicity: H ARGP820101 D Hydrophobicity index (Argos et al., 1982) R PMID:7151796 A Argos, P., Rao, J.K.M. and Hargrave, P.A. T Structural prediction of membrane-bound proteins J Eur. J. Biochem. 128, 565-575 (1982) C JOND750101 1.000 SIMZ760101 0.967 GOLD730101 0.936 TAKK010101 0.906 MEEJ810101 0.891 ROSM880104 0.872 CIDH920105 0.867 LEVM760106 0.865 CIDH920102 0.862 MEEJ800102 0.855 MEEJ810102 0.853 ZHOH040101 0.841 CIDH920103 0.827 PLIV810101 0.820 CIDH920104 0.819 LEVM760107 0.806 NOZY710101 0.800 GUYH850103 -0.808 PARJ860101 -0.835 WOLS870101 -0.838 BULH740101 -0.854 I A/L R/K N/M D/F C/P Q/S E/T G/W H/Y I/V 0.61 0.60 0.06 0.46 1.07 0. 0.47 0.07 0.61 2.22 1.53 1.15 1.18 2.02 1.95 0.05 0.05 2.65 1.88 1.32 //We’re interested in those values along the bottom. These are the feature Infinity Stones I collected for our dataset Gauntlet: Hydrophobicity: a measure of the tendency of that side chain to be located in the interior of a native protein molecule. Normalized to be unitless. Note: In the original paper, the units were ΔFₜ — for the transfer of the amino acid side chain from organic solvent to water at 25° in cal/mole. Here, they are all made positive and normalized to a value of 1.0, which means they adjusted the scale so that the average value is 1.0. Steric parameter: the tendency of amino acids to be found at the surface of globular proteins as a function of side-chain structure. Unitless, but calculated based on van der Waals radii. Residue volume: The space a residue occupies. Units are Ų. Polarizability: a measure of how easily an amino acid’s electron cloud will form dipoles in response to external forces. Unitless, a relative measure. Average relative probability of helix: the relative helix residue probability in 47 proteins. Unitless, relative measure. Average relative probability of beta sheet: the relative beta sheet residue probability in 47 proteins. It is also a unitless, relative measure. Isoelectric point: the pH at which an amino acid is electrically neutral. Units are pH. The Biology: Why these features? Here’s an example: we might expect binding residues to be on the surface of the protein, where it can easily bind to other peptides — so a measure like steric parameter is useful. Using AAindex1, I load all of these indices into a script aaindex.py full of dictionaries and a function to access them: # aaindex.py def feature_vector(seq: str, feature_type: dict): ''' Returns a list of feature values corresponding to each residue in sequence, according to AAindex1. Feature types include: - Hydrophobicity - Steric parameter - Residue volume - Polarizability - Average relative probability of helix - Average relative probability of beta - Isoelectric point ''' return [feature_type[aa] for aa in seq] hydrophobicity = { 'A': 0.61, ..., 'V': 1.32 }Next, I extract this information from every sequence: # gendata.py print('Adding residue-level data from AAindex1...') ## Add rich data from AAindex1 features = { 'Hydrophobicity': hydrophobicity, 'Steric Parameter': steric_parameter, 'Volume': residue_volume, 'Polarizability': polarizability, 'Helix Probability': average_relative_probability_of_helix, 'Beta Probability': average_relative_probability_of_beta_sheet, 'Isoelectric Point': isoelectric_point } for feature_name, feature_function in features.items(): peptide_list[f'Peptide {feature_name}'] = peptide_list['Peptide Sequence'].apply(lambda x: feature_vector(x, feature_function)) peptide_list[f'Protein {feature_name}'] = peptide_list['Protein Sequence'].apply(lambda x: feature_vector(x, feature_function)) print('\\\\033[1mAAindex features added.\\\\033[0m')Note: since completing this piece, I’ve discovered a Python package for accessing all data from the AAindex. Which means I didn’t need to write this script from scratch. Yes, I have feelings about this. No, I will not be discussing them.PSI-BLASTNext, let’s use PSI-BLAST to get residue-level conservation scores.PSI-BLAST (Position-Specific Iterative Basic Local Alignment Search Tool) is a sequence alignment tool that lets us build a Position Specific Scoring Matrix (PSSM) from a sequence.A PSSM tells us how conserved a given residue is at a given position in the sequence. High score = highly conserved. The Biology: Why do we want this information? Binding motifs are typically highly conserved — alterations could mean loss of binding function, stymying organism survival.Here’s an example for the same protein we looked at earlier, 1A03: Last position-specific scoring matrix computed, weighted observed percentages rounded down, information per position, and relative weight of gapless real matches to pseudocounts A R N D C Q E G H I L K M F P S T W Y V A R N D C Q E G H I L K M F P S T W Y V 1 M -3 -4 -4 -5 -4 -3 -4 -5 -4 -1 0 -4 10 -2 -5 -4 -3 -4 -3 -1 0 0 0 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 2.17 0.45 2 A 1 -3 0 -2 1 -2 -1 1 -3 -3 -2 -2 -3 -4 6 0 -1 -4 -4 -3 14 0 5 0 4 0 3 12 0 0 5 0 0 0 48 7 2 0 0 0 0.92 0.21 3 S -1 0 -1 -2 1 -2 -2 -2 -2 -1 -3 -1 -2 -3 -2 4 4 -4 -3 -2 2 6 0 1 3 0 1 1 0 4 1 2 0 0 0 40 37 0 0 1 0.61 0.25 4 P -2 -2 -2 3 -5 4 3 -3 -2 -4 -3 1 -3 -5 4 -1 -2 -5 -4 -4 0 0 0 15 0 23 22 0 0 0 3 9 0 0 25 1 0 0 0 0 0.80 0.28 .... 86 E -2 -1 0 3 -4 0 4 -3 2 -3 -2 3 -3 -4 0 -2 -2 -4 -1 -3 0 0 3 20 0 0 33 0 5 1 5 26 0 0 5 0 0 0 2 0 0.55 0.20 87 A -1 -1 0 0 -3 2 0 -3 2 -1 -2 2 -2 3 -3 0 -1 -1 3 -1 4 0 3 6 0 11 5 0 6 4 0 17 0 14 0 9 0 0 15 6 0.21 0.12 88 L -2 -3 -3 -4 -3 -3 -4 -4 -3 1 3 -3 3 5 -4 1 -1 -1 2 -1 0 0 0 0 0 0 0 0 0 7 28 0 9 34 0 14 2 0 6 0 0.54 0.19 89 K -3 0 -1 0 -5 2 2 -3 9 -5 -4 2 -3 -3 -3 -2 -3 -4 0 -4 0 2 0 5 0 7 11 0 62 0 0 12 0 0 0 0 0 0 0 0 1.18 0.25 90 G 0 -2 3 -1 -3 -2 -2 5 -2 -4 -4 -2 -3 -4 -3 0 -2 -3 -3 -3 6 0 23 0 0 0 0 72 0 0 0 0 0 0 0 0 0 0 0 0 0.87 0.08 K Lambda Standard Ungapped 0.1364 0.3155 Standard Gapped 0.0519 0.2670 PSI Ungapped 0.1697 0.3169 PSI Gapped 0.0519 0.2670NCBI provides PSI-BLAST as a command-line tool, so we’ll write some code that invokes it for all of our sequences. We’ll create a temporary FASTA file with the sequence so we can input it into PSI-BLAST. swissprot, which is a path to the SwissProt database (imported from paths, licensed under CC BY 4.0, and can be downloaded from NCBI. You can use other databases, but this one was smaller and easier to manage. # helpers.py def get_pssm_profile(sequence: str) -> pd.DataFrame: ''' Uses blast+ psiblast to generate PSSM profile from a temporary fasta file. ''' with tempfile.NamedTemporaryFile(suffix='.fa', delete=False) as fasta_file: fasta_file.write(f'>tmp\\n{sequence}'.encode('utf-8')) fasta_path = fasta_file.name pssm_fd, pssm_path = tempfile.mkstemp(suffix='.pssm') os.close(pssm_fd) try: subprocess.run( ['psiblast', '-query', fasta_path, '-db', swissprot, '-num_iterations', '3', '-evalue', '0.001', '-out_ascii_pssm', pssm_path], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=True ) with open(pssm_path, 'r') as file: lines = file.readlines() # Skip the header lines and parse the matrix pssm_data = [] for line in lines[3:len(sequence) + 3]: parts = line.strip().split() scores = parts[1:22] # First 20 columns are scores for each amino acid pssm_data.append(scores) columns = ['AA'] + list('ARNDCQEGHILKMFPSTWYV') df_pssm = pd.DataFrame(pssm_data, columns=columns) finally: os.remove(fasta_path) os.remove(pssm_path) return df_pssm.T Now, let’s apply this function to our dataset: # gendata.py print('Now generating and filtering PSSMs...') ## Add PSSM profiles into dataframe peptide_list['Peptide PSSM'] = peptide_list['Peptide Sequence'].apply(get_pssm_profile) peptide_list['Protein PSSM'] = peptide_list['Protein Sequence'].apply(get_pssm_profile) print('\\\\033[1mPSSMs generated.\\\\033[0m') 5. Structural information using BioPythonNow that we’ve extracted our information from the sequences, let’s take a closer look at the 3D models.Secondary structure codes, half-sphere exposure, and from DSSPThe Dictionary of Secondary Structure in Proteins (DSSP) determines the most likely secondary structure assignment of a residue given a 3D structure. While the original paper uses SPIDER2 to predict this, we can use the PDB itself.# helpers.py def hse_and_dssp(pdb_file: str): ''' Get the HSE and DSSP codes of a PDB's residues. ''' with warnings.catch_warnings(): warnings.simplefilter(action='ignore', category=FutureWarning) # Initialize PDBParser pdb_parser = PDBParser() structure = pdb_parser.get_structure('structure', pdb_file) hse = HSExposure.HSExposureCA(structure) dssp = DSSP(structure[0], pdb_file) # Half-sphere exposure values hse_up, hse_down, pseudo_angle = zip(*[(res[1][0], res[1][1], res[1][2]) for res in hse.property_list]) # Extract DSSP values into separate lists ss, asa, phi, psi = zip(*[(res[2], res[3], res[4], res[5]) for res in dssp.property_list]) print(f'\\\\rData acquired for {pdb_file}') return hse_up, hse_down, pseudo_angle, ss, asa, phi, psi def safe_hse_and_dssp(pdb_file): ''' Helper function to handle errors and apply `hse_and_dssp`. ''' # Empty list to store error-producing filenames error_files = [] try: return hse_and_dssp(pdb_file) except Exception as e: print(f'Error processing file: {pdb_file} - {e}') error_files.append(pdb_file) return [None] * 7 # Return a list of None values to match the expected output structureLet’s break down each of these features: Secondary structure ss: These are DSSP structure assignment codes. Relative accessible surface area asa: a measure of how accessible a residue in the sequence is by its surrounding solvent, normalized by the maximum possible ASA for that residue. (Unitless, but the maximum ASA is in Ų) phi angles: the angle of rotation about the (nitrogen)-(alpha carbon) bond. (In degrees). psi angles: the angle of rotation about the (alpha carbon)-(carbonyl carbon) atom. (In degrees). Half-sphere exposure, up and down hse_up, hse_down: measures of how buried a residue is in the overall protein structure using the number of Cɑs in the up- and down-facing hemispheres of the amino acid. Units are integer counts. Pseudo angles pseudo_angle: When calculating HSE, we also get the angle between three adjacent Cɑs. This pseudo-angle helps divide the amino acid into hemispheres and can tell us the shape of the protein backbone at a particular residue. The unit is in degrees. # gendata.py print('Adding HSE, ASA, and DSSP codes...') ## Add HSE, ASA, DSSP codes, etc.peptide_list[['Peptide HSE Up', 'Peptide HSE Down', 'Peptide Pseudo Angles', 'Peptide SS', 'Peptide ASA', 'Peptide Phi', 'Peptide Psi']] = peptide_list['Peptide Path'].apply(lambda x: pd.Series(safe_hse_and_dssp(x)))peptide_list[['Protein HSE Up', 'Protein HSE Down', 'Protein Pseudo Angles', 'Protein SS', 'Protein ASA', 'Protein Phi', 'Protein Psi']] = peptide_list['Protein Path'].apply(lambda x: pd.Series(safe_hse_and_dssp(x)))Before we wrap up, there are a few more housekeeping steps we need to take care of: # One-hot encoding function def one_hot_encode_array(ss_array): ss_array = literal_eval(str(ss_array)) length = len(ss_array) encoding = {code: [0] * length for code in dssp_codes} for i, code in enumerate(ss_array): encoding[code][i] = 1 return encoding def one_hot_encode_row(row): pep_encoded = one_hot_encode_array(row['Peptide SS']) prot_encoded = one_hot_encode_array(row['Protein SS']) new_data = {} for code in dssp_codes: new_data[f'Peptide SS {code}'] = pep_encoded[code] new_data[f'Protein SS {code}'] = prot_encoded[code] return pd.Series(new_data) def extend_hse(hse: str) -> str: \"\"\" Extends a HSE to the full length of the peptide. \"\"\" hse = list(literal_eval(str(hse))) hse = [hse[0]] + hse + [hse[-1]] return hse ## One-hot encode SS codes ss_columns = peptide_list.apply(one_hot_encode_row, axis=1) peptide_list = pd.concat([peptide_list, ss_columns], axis=1) print('\\\\033[1mHSE, ASA, and DSSP codes added and encoded.\\\\033[0m') ## Extend HSE and Pseudo Angles to match peptides length. Achieved by duplicating terminal values. peptide_list['Protein HSE Up'] = peptide_list['Protein HSE Up'].apply(extend_hse) peptide_list['Protein HSE Down'] = peptide_list['Protein HSE Down'].apply(extend_hse) peptide_list['Protein Pseudo Angles'] = peptide_list['Protein Pseudo Angles'].apply(extend_hse) peptide_list['Peptide HSE Up'] = peptide_list['Peptide HSE Up'].apply(extend_hse) peptide_list['Peptide HSE Down'] = peptide_list['Peptide HSE Down'].apply(extend_hse) peptide_list['Peptide Pseudo Angles'] = peptide_list['Peptide Pseudo Angles'].apply(extend_hse) print('\\\\033[1mHSE data extended.\\\\033[0m') Since this data is going to be used for ML/DL projects, we’ll save ourselves some work by one-hot encoding the secondary structure columns. HSEup, HSEdown, and pseudo angles all need three residues, so the terminal residues are left without any values. We’ll shortcut around this — extending the terminal edges by duplicating the first and last valuesFinally, we’ll reduce the table so proteins and peptides are no longer sharing rows before converting the whole dataset into a tabular array matching our initial specifications: # gendata.py ## Reduce to one peptide/protein per row peptide_cols = [col for col in peptide_list.columns if 'Peptide' in col] # Columns we want to keep in pep_data peptide_cols.remove('Peptide Length') protein_cols = [col for col in peptide_list.columns if 'Protein' in col] # Columns we want to keep in pro_data pep_data = peptide_list[peptide_cols] # Select the peptide data pro_data = peptide_list[protein_cols] # Select the protein data # Combine the two dataframes combined_data = pd.DataFrame(np.vstack([pep_data, pro_data.values])) combined_data.columns = pro_data.columns # Replace the original peptide_list with the combined data peptide_list = combined_dataNext, we’ll define our table-making function: # helpers.py def make_tabular_dataset(row: pd.Series) -> pd.DataFrame: ''' `make_tabular_dataset` is designed to be applied to a row of the input DataFrame to create a feature array. ''' feature_dict = row.to_dict() # get sequence and PSSM sequence = feature_dict[f'Protein Sequence'] pssm = feature_dict[f'Protein PSSM'] # remove the first row of the PSSM, which is the sequence pssm = pssm.iloc[1:] pssm = pssm.values # get the binding indices list binding_indices_dummy = [0] * len(sequence) binding_indices = feature_dict[f'Protein Binding Indices'] for i in range(len(sequence)): if i+1 in literal_eval(binding_indices): binding_indices_dummy[i] = 1 # add the sequence as the first element of the array arr = [] arr.append([char for char in sequence]) # remove these columns from the dictionary; all are either unneeded or have been processed remove = ['Unnamed: 0', 'PDB ID', 'Number of Atoms in Protein', 'Protein Chain ID', 'Protein Path', 'Number of Atom Contacts', 'Resolution', 'Molecular Type', 'Protein Path', 'Protein Sequence', 'Protein Binding Indices', 'Protein SS', 'Protein PSSM'] # many of the features are stored as strings, so we need to convert them to lists use_feature_dict = {} for k, v in feature_dict.items(): if k not in remove: try: use_feature_dict[k] = literal_eval(str(v)) except (ValueError, SyntaxError) as e: print(f\"Error parsing key '{k}' with value '{v}': {e}\") # stack all the features for _, value in use_feature_dict.items(): value_array = [residue_value for residue_value in value] arr.append(value_array) # add the feature to the array for aa_row in pssm: arr.append(aa_row) arr.append(binding_indices_dummy) # transpose the array and add the column names arr = pd.DataFrame(arr).T columns = ['AA'] + list(use_feature_dict.keys()) + list('ARNDCQEGHILKMFPSTWYV') + ['Binding Indices'] if len(arr.columns) == len(columns): arr.columns = columns else: print(\"Column length mismatch\") print(\"Arr shape:\", arr.shape) print(\"Columns length:\", len(columns)) return pd.DataFrame(arr) # gendata.py ## Create tabular dataset, 1 row per residue list_of_feature_arrays = [] for i, row in peptide_list.iterrows(): \tarr = make_tabular_dataset(row) \tlist_of_feature_arrays.append(arr) \tprint(f'\\\\r{i+1}/{peptide_list.shape[0]}', end='') print('\\\\033[1m\\\\nConverted.\\\\033[0m') export = pd.concat(list_of_feature_arrays) export = export.dropna() export = export.reset_index(drop=True) export.to_csv(peppi_data_csv, index=False)But let’s not forget — we also wanted to generate an image dataset. Here’s how we do that with our tabular dataset.Optional: Generating an image datasetWe need to write functions that: Generates a sliding window over each sequence, seven residues wide. Turns those windows into labeled images. # helpers.py def window_maker(sequence_array: pd.DataFrame) -> List[pd.DataFrame]: ''' Takes a sequence array and returns a list windowed arrays by sliding a window over the input feature array. The window is centered on the residue of interest. ''' k = 7 # window size windows = [] sequence_array = sequence_array.T.reset_index(drop=True) sequence_length = sequence_array.shape[1] for i in range(sequence_length): # N-terminal case if i < 3: right = sequence_array.iloc[:, 0:i+4] if i == 0: left = right.iloc[:, :-4:-1] elif i == 1: left = right.iloc[:, 1::-1] elif i == 2: left = right.iloc[:, 1] window = pd.concat([left, right], axis = 1) # C-terminal case elif i >= sequence_length - 3: left = sequence_array.iloc[:, i-3:] if i == sequence_length - 3: right = sequence_array.iloc[:, i+1] elif i == sequence_length - 2: right = sequence_array.iloc[:, i-2:i].iloc[:, ::-1] elif i == sequence_length - 1: right = sequence_array.iloc[:, i-3:i].iloc[:, ::-1] window = pd.concat([left, right], axis = 1) # Standard case else: window = sequence_array.iloc[:, i-3:i+4] windows.append(window) return windows def create_images(window: pd.DataFrame, name: str): ''' Helper function that accepts a window DataFrame and converts it into a CNN-friendly image. ''' window = window.T.reset_index(drop=True) # transpose the array and reset the index window = window.apply(pd.to_numeric, errors='coerce') # convert to numeric window = window.to_numpy() # convert to numpy array window_normalized = window * 255 # scale to 0-255 window_uint8 = window_normalized.astype('uint8') img = Image.fromarray(window_uint8) img.save(name) print(f'\\\\rCreated image: {name}.', end='') def process_images(list_of_feature_arrays: List[pd.DataFrame], binding_path: str, nonbinding_path: str): ''' Takes a list sequence feature arrays and uses create_images to turn valid ones into images in the appropriate folder. ''' os.makedirs(binding_path, exist_ok=True) os.makedirs(nonbinding_path, exist_ok=True) name_index = 0 for arr in list_of_feature_arrays: residues = arr.pop('AA') binding_indices = arr.pop('Binding Indices') binding_indices = binding_indices.to_list() windows = window_maker(arr) for i, window in enumerate(windows): if not window.isna().any().any(): name_index += 1 folder = binding_path if binding_indices[i] == 1 else nonbinding_path name = f'{folder}/{name_index}.jpg' create_images(window, name) else: continue print('\\\\n')There’s a lot to unpack here, so let’s take it function by function: window_maker : notice how each element in list_of_feature_arrays is a dataframe corresponding to one particular sequence? window_maker runs a sliding window over it to generate residue-specific arrays. create_images: this function accepts those windows as input, normalizes the values, and converts them into images. process_images: here, we run the earlier two functions and save the images to the appropriate folder based on the label. # gendata.py ## Optional step to create images:if args.images: print('\\\\033[1m\\\\nNow creating images...\\\\033[0m') process_images( list_of_feature_arrays, binding_path=args.binding_path, nonbinding_path=args.nonbinding_path )And there we have it! 800,000+ tabular records of rich, residue-level data. 800,000+ images of residues ready for classifier training. Encoded and transformed to be ML/DL friendly. Behold — the PepBDB-ML database.For the data-curious among us — don’t worry. Here are some visualizations to explore the dataset now that we have it available:The Results: Visualizing the PepBDB-ML databaseAmino acid distributions ![Figure 1. Bar chart of amino acid counts in PepBDB-ML. By Author](https://cdn-images-1.medium.com/max/2000/1*ch6d7U1KXSY5sZGQWkqqbw.png) Figure 1. Bar chart of amino acid counts in PepBDB-ML.There isn’t too much to note here, apart from the curious observation that Leucine outnumbers every other amino acid in the database. ![Figure 2. Bar chart of amino acid distributions in binding (left) and non-binding residues (right). By Author](https://cdn-images-1.medium.com/max/2000/1*rNMg8NvXDalEcU-31bD6MA.png) Figure 2. Bar chart of amino acid distributions in binding (left) and non-binding residues (right).Splitting our amino acid counts by binding vs. nonbinding residues shows us the relative proportion of AAs differs ever so slightly. Here’s a view of both in the same plot: ![Figure 3. Bar chart of amino acid percentages by class in PepBDB-ML. By Author](https://cdn-images-1.medium.com/max/2000/1*lpnFm06hwr6-cM1Tw5hpMA.png) Figure 3. Bar chart of amino acid percentages by class in PepBDB-ML.Takeaways: The residue makeup of binding and nonbinding residues might differ slightly. (In other breaking news, the sky is blue). Correlation mapping ![Figure 4. Correlation heatmap of features in PepBDB-ML. By Author](https://cdn-images-1.medium.com/max/2742/1*2KW9mkpGbjgxbcbEUBXT_w.png) Figure 4. Correlation heatmap of features in PepBDB-ML.A correlation heatmap helps us visualize correlations between variables in our dataset. The colours and labels tell us the strength and directions of those correlations.Some takeaways: A strong relationship between residue polarizability and volume (0.96). This makes a lot of sense — large amino acids have more electrons around them, which makes them more polarizable. A weak relationship between psi angle and the residue being involved in a helix. Psi angles fall between -50–100° in helices, which may explain part of what we see here. An interesting point seems to be a weakly positive relationship between relative beta-sheet folding probability and substitutability with Isoleucine. Feature distributions ![Figure 5. Distributions of 41 features found in PepBDB-ML. By Author](https://cdn-images-1.medium.com/max/3002/1*rL-BeJZjiMjtAfnxRNKgFA.png) Figure 5. Distributions of 41 features found in PepBDB-ML.Some interesting callouts: Pseudoangles have a bimodal distribution. PSI-BLAST substitution scores are primarily normally distributed, but… Tryptophan (W) skews low — meaning most sequences don’t want to change partners with W. Since tryptophan is the largest amino acid (204.23 g/mol), its bulk can cause steric hindrance should it be swapped with smaller residues. Now, let’s compare distributions between binding and non-binding residues: ![Figure 6. Distributions of 41 features found in PepBDB-ML by class. By Author](https://cdn-images-1.medium.com/max/3020/1*fPzkkcQQRdKFv5EBQucAgg.png) Figure 6. Distributions of 41 features found in PepBDB-ML by class.The number of nonbinding residues far outnumbers binding residues, so it’s a little tricky to parse in this visualization — but feature distributions appear quite similar between samples.Does this mean there isn’t a discernible difference between our samples? This could cause some issues if we plan on using this dataset for training classifiers. Let’s do some quick statistical analysis:Statistical tests Small p-values for both the KS and U-test tell us there are statistically significant differences between the distributions of nonbinding and binding samples. However, a lot of this is due to the difference in sizes of both samples. Looking at the effect size (Cohen’s d) we see these differences are quite small. To truly parse any meaningful difference between the groups, we’ll have to leverage a machine learning model. The Discussion: Key takeaways and learningsDeveloping this dataset was step 1 in trying to reimplement the code from the paper I mentioned at the start.I (naively) assumed the data construction would be the easy part. In reality, the bulk of my time was spent wrangling the myriad of enrichment tools and databases, accounting for edge cases in all of them, and formatting.But it paid off.The CNN I trained on this data outperforms the original by 12 pts. on binding site detections (79%) — but I’ll save that for a future article.Key takeaways: A model is only as good as the data that it’s trained on. Garbage in, garbage out, as they say. In any machine learning project, the hardest part is often data preprocessing. Put effort into curating better, higher-quality datasets, and you’ll reap the rewards. You can download the full dataset at my GitHub or my website. Running the data generation scripts can take some time, so ensure you have enough computing power or access to an HPC.Let me know if you have any feedback on improving — thanks for reading!", "content_html": "

\"Visualization

\"GitHub\"License\"\"Stars\"

Table of Contents

90% of the time I tell people I work with proteins, they assume I’m trying to get jacked.

And okay, fine, they’re right.

But really, I’m talking about biological proteins— incredible molecular machines that make it possible for your body to function.

\"Photo

Proteins are involved in vital metabolic processes — including getting that summer body — and peptide-protein interactions (PepPIs) are among the most important.

(Peptide = short protein segments).

Recently, I built a dataset I could use to study PepPIs. And by the end of this article, you will have too.

Here’s a quick peek at a few things I discovered in what I’m calling the PepBDB-ML database:

But I’m getting ahead of myself —

Let’s start with some background.

Background: CNNs can predict binding sites

Recently, I read a paper that used CNNs to predict peptide-protein binding sites from a sequence.

\"From

I thought this approach was fascinating, but I saw an opportunity for improvement: residue secondary structure tags, angles, and accessible surface area were all predicted using SPIDER2 — but what if I just got the actual data from experimental structures instead?

That’s precisely what I did using the PepBDB, a structural database of 3D peptide-receptor complexes (all structures are curated from the RCSB PDB, which provides data under license CC0).

\"Screenshot

By using all the information applied in Visual, along with some structure-specific features, I could:

(Two truths, one lie).

So that is precisely what I set out to do with PepBDB-ML

The Rationale: But wait — does the data really matter that much?

Well, there’s one key reason I think focusing on data is one of the highest-leverage activities in AI: data is our greatest constraint.

Sure, compute is critical — but the data hydra has three heads:

1. Availability

Heard about the recent content licensing deal between OpenAI and News Corp, Vox Media, and The Atlantic? The agreement with News Corp alone is believed to be worth over $250 million.

While a lot of AI training data is either unprotected or open-source, the good stuff is often unavailable, expensive, or hard to find. Which leads to the next head…

2. Quality

Let’s use an example.

If LLMs right now are really good at generating text, why not use them to create more training data? Unfortunately, making AI eat its own tail causes something called Model Autophagy Disorder, or MAD — and yes, they came up with the acronym before deciding what it stood for.

Bottom line? Synthetic data can only take us so far. In the world of biology, that means we need reliable, experimental data (like the structures from PepBDB).

3. Interpretability

In biology specifically, human-readable data is often the most helpful. It allows us to unpack machine learning models and see things like:

For example, when it comes to peptide-protein binding, what’s more impactful — hydrophobicity or half-sphere exposure? Without labels, we can’t answer that question.

Documented data means that models and humans can both learn more about biology.

Key Points:

Now that you’re convinced we’re not wasting time, let’s get into the meat of this project: building the dataset.

The Process: Building the PepBDB-ML dataset

Before we begin, let’s decide what we want our dataset to look like.

With this plan in mind, we can focus our efforts on the tabular dataset. Since the PepBDB database contains structures, we can use structural and sequence information. Here’s how we’ll approach this:

  1. Initial filtering

  2. Sequence extraction + filtering

  3. Binding residue determination with Prodigy

  4. Sequence level information (AAindex, PSI-BLAST)

  5. Structural information from BioPython (DSSP, HSE, Pseudo-angles)

There’s going to be a lot of code coming up, so buckle up.

Before we begin, we’ll start with our imports:

    # gendata.py        import pandas as pd    import numpy as np    import argparse    import os    import pickle        from aaindex import *    from helpers import *    from paths import *

What are those ones at the bottom? Don’t worry — we’ll get to those soon.

(Heads up — since we’re going to be jumping between a few scripts, I’ll include the filename at the top of each code block).

1. Initial filtering

The PepBDB offers the peptidelist.txt file as part of its download. This file provides information about each peptide-protein pair in the repository.

    # gendata.py        ## Load the peptidelist.txt file    peptide_list = pd.read_csv(peptide_list_txt, sep='\\\\\\\\s+', header=None)        headers = ['PDB ID', 'Peptide Chain ID', 'Peptide Length', 'Number of Atoms in Peptide',               'Protein Chain ID', 'Number of Atoms in Protein',               'Number of Atom Contacts', 'unknown1', 'unknown2', 'Resolution', 'Molecular Type']    peptide_list.columns = headers        # 'pepbdb' below is the path to the PepBDB directory, imported from `paths`    peptide_list['Peptide Path'] = pepbdb + peptide_list['PDB ID'] + '_' + peptide_list['Peptide Chain ID'] + '/peptide.pdb'    peptide_list['Protein Path'] = pepbdb + peptide_list['PDB ID'] + '_' + peptide_list['Peptide Chain ID'] + '/receptor.pdb'    peptide_list.reset_index(drop=True)        ## Filter out nucleic acids, low-resolution, and small peptides    peptide_list = peptide_list[peptide_list['Molecular Type'] != 'prot-nuc']    peptide_list = peptide_list[peptide_list['Resolution'] < 2.5]    peptide_list = peptide_list[peptide_list['Peptide Length'] >= 10]    print('\\\\033[1mInitial filtering done.\\\\033[0m')

We’ll do some very simple filtering to begin:

In the next step, we’ll use BioPython to extract the sequences from each of our remaining models. Sequence extraction can take some time, which is why I’ve chosen to do it after some initial cleanup.

2. Sequence extraction using BioPython

PepBDB provides 3D models but not direct sequences. Fortunately, these are relatively simple to extract using BioPython.

To keep the code clean, I’ll create a helper function in a separate file called helpers.py:

    # helpers.py        def extract_sequence(pdb_filename: str) -> str:        '''        Helper function that extracts the sequence from a PDB file.        '''        parser = PDBParser()        structure = parser.get_structure('X', pdb_filename)                sequence = ''        contains_unk = False                for model in structure:            for chain in model:                residues = chain.get_residues()                for residue in residues:                    if residue.get_resname() == 'HOH': # ignoring water                        continue                    if residue.get_resname() == 'UNK':                        sequence += 'X'  # add X for each 'UNK' residue                    else:                        sequence += seq1(residue.get_resname()) # seq1 converts 3-letter code to 1-letter code            return sequence

Then, I’ll import it into our data generation script and apply it to our dataset:

    # gendata.py        peptide_list['Peptide Sequence'] = peptide_list['Peptide Path'].apply(extract_sequence)    peptide_list['Protein Sequence'] = peptide_list['Protein Path'].apply(extract_sequence)        print('\\\\\\\\033[1mSequences extracted.\\\\\\\\033[0m')

Now that we have sequences for each PDB structure, let’s remove the problematic ones (sequences with non-standard amino acids):

    # gendata.py        peptide_list = peptide_list[~peptide_list['Peptide Sequence'].str.contains('[UOBZJX\\\\\\\\\\\\\\\\*]', regex=True)]    peptide_list = peptide_list[~peptide_list['Protein Sequence'].str.contains('[UOBZJX\\\\\\\\\\\\\\\\*]', regex=True)]        peptide_list = peptide_list.drop_duplicates(subset=['Peptide Sequence', 'Protein Sequence'])

\"Screenshot

3. Binding sequences determination with PRODIGY

PRODIGY (PROtein binDIng enerGY prediction) is a tool for predicting binding affinities for protein structures from 3D data.

Part of that involves identifying residues in contact with each other — i.e., involved in binding.

I used PRODIGY on every PDB in the PepBDB leftover from the last preprocessing step. The program outputs a .icfile — a simple text file that lists the residues involved in binding.

Here’s an example for calcium-bound cyclin (1A03):

      GLY    78   A   GLN    71   B      ILE    13   A   TYR    84   B      LYS    40   A   MET     1   B      GLU    86   A   HIS    27   B      LEU    88   A   LEU    12   B      ALA     8   A   ILE    15   B      MET     1   A   LYS    40   B      ALA    87   A   ILE    13   B      PHE    70   A   LEU    77   B      GLN    71   A   MET    82   B      ALA     8   A   ALA     8   B      ILE    74   A   LEU    77   B      LYS    89   A   LYS    26   B

Processing this data gives us two lists of indices, each referring to a binding residue.

The peptides and receptors in each row are available as distinct PDBs, i.e. their models contain one chain each. For our purposes, PRODIGY needs a complex (2 chains), so we’ll make a temporary file and process the output:

    # helpers.py        def label_residues(peptide_path: str, protein_path: str) -> List:        # creating a temporary file with the peptide and protein        with tempfile.NamedTemporaryFile(suffix='.pdb', delete=False) as temp_file:            output_path = temp_file.name            subprocess.run(['cat', peptide_path, protein_path], stdout=temp_file)            # ensuring the temp file is properly closed before using it in subprocess        temp_file.close()                # obtaining binding residues using PRODIGY        try:            subprocess.run(['prodigy', '-q', '--contact_list', output_path], check=True)        except subprocess.CalledProcessError as e:            print(f\"An error occurred while processing the files: {peptide_path} & {protein_path}\")            print(f\"Error: {e}\")                # the .ic file will have the same root name as the input file        ic_file_path = output_path.replace('.pdb', '.ic')            # check if the .ic file exists before reading        if not os.path.exists(ic_file_path):            raise FileNotFoundError(f\"{ic_file_path} not found after running PRODIGY.\")                # are you actually reading these comments? kudos! if yes — reply with a 😼        contacts = pd.read_csv(ic_file_path, sep='\\s+', header=None)        contacts.columns = ['peptide_residue', 'peptide_index', 'peptide_chain', 'protein_residue', 'protein_index', 'protein_chain']                peptide_binding_residues = sorted(list(contacts['peptide_index'].unique()))        protein_binding_residues = sorted(list(contacts['protein_index'].unique()))                os.remove(output_path)                return peptide_binding_residues, protein_binding_residues

Now, let’s apply it to our dataset:

    # gendata.py        print('Running PRODIGY to determine binding contacts...')        ## determine binding residues from peptide-protein complexes    peptide_list['Peptide Binding Indices'] = np.nan    peptide_list['Protein Binding Indices'] = np.nan        for index, row in peptide_list.iterrows():        peptide_path = row['Peptide Path']        protein_path = row['Protein Path']        peptide_binding_positions, protein_binding_positions = label_residues(peptide_path, protein_path)        peptide_list.at[index, 'Peptide Binding Indices'] = str(peptide_binding_positions)        peptide_list.at[index, 'Protein Binding Indices'] = str(protein_binding_positions)    print('\\\\\\\\033[1mBinding indices identified.\\\\\\\\033[0m')

\"Screenshot

The Biology: You might be wondering — why build this dataset if we can use PRODIGY to get binding sites? Well, PRODIGY generally needs two chains in complex. Our dataset is aimed at helping ML/DL models predict binding sites with just one model chain.

4. Adding sequence level information with AAindex and PSI-BLAST

AAindex

We’ve now got a list of protein sequences. We can enrich our dataset by adding publically available, residue-specific information from the Amino Acid Index Database — a cornucopia of known, reference amino acid features.

For example, here’s hydrophobicity:

    H ARGP820101    D Hydrophobicity index (Argos et al., 1982)    R PMID:7151796    A Argos, P., Rao, J.K.M. and Hargrave, P.A.    T Structural prediction of membrane-bound proteins    J Eur. J. Biochem. 128, 565-575 (1982)    C JOND750101    1.000  SIMZ760101    0.967  GOLD730101    0.936      TAKK010101    0.906  MEEJ810101    0.891  ROSM880104    0.872      CIDH920105    0.867  LEVM760106    0.865  CIDH920102    0.862      MEEJ800102    0.855  MEEJ810102    0.853  ZHOH040101    0.841      CIDH920103    0.827  PLIV810101    0.820  CIDH920104    0.819      LEVM760107    0.806  NOZY710101    0.800  GUYH850103   -0.808      PARJ860101   -0.835  WOLS870101   -0.838  BULH740101   -0.854    I    A/L     R/K     N/M     D/F     C/P     Q/S     E/T     G/W     H/Y     I/V        0.61    0.60    0.06    0.46    1.07      0.    0.47    0.07    0.61    2.22        1.53    1.15    1.18    2.02    1.95    0.05    0.05    2.65    1.88    1.32    //

We’re interested in those values along the bottom. These are the feature Infinity Stones I collected for our dataset Gauntlet:

Using AAindex1, I load all of these indices into a script aaindex.py full of dictionaries and a function to access them:

    # aaindex.py        def feature_vector(seq: str, feature_type: dict):        '''        Returns a list of feature values corresponding to each        residue in sequence, according to AAindex1.                Feature types include:        - Hydrophobicity        - Steric parameter        - Residue volume        - Polarizability        - Average relative probability of helix        - Average relative probability of beta        - Isoelectric point        '''        return [feature_type[aa] for aa in seq]        hydrophobicity = {        'A': 0.61,      ...,        'V': 1.32    }

Next, I extract this information from every sequence:

    # gendata.py        print('Adding residue-level data from AAindex1...')        ## Add rich data from AAindex1    features = {     'Hydrophobicity': hydrophobicity,      'Steric Parameter': steric_parameter,      'Volume': residue_volume,      'Polarizability': polarizability,      'Helix Probability': average_relative_probability_of_helix,      'Beta Probability': average_relative_probability_of_beta_sheet,      'Isoelectric Point': isoelectric_point        }    for feature_name, feature_function in features.items():     peptide_list[f'Peptide {feature_name}'] = peptide_list['Peptide Sequence'].apply(lambda x: feature_vector(x, feature_function))      peptide_list[f'Protein {feature_name}'] = peptide_list['Protein Sequence'].apply(lambda x: feature_vector(x, feature_function))    print('\\\\033[1mAAindex features added.\\\\033[0m')

\"Screenshot

Note: since completing this piece, I’ve discovered a Python package for accessing all data from the AAindex. Which means I didn’t need to write this script from scratch. Yes, I have feelings about this. No, I will not be discussing them.

PSI-BLAST

Next, let’s use PSI-BLAST to get residue-level conservation scores.

PSI-BLAST (Position-Specific Iterative Basic Local Alignment Search Tool) is a sequence alignment tool that lets us build a Position Specific Scoring Matrix (PSSM) from a sequence.

A PSSM tells us how conserved a given residue is at a given position in the sequence. High score = highly conserved.

The Biology: Why do we want this information? Binding motifs are typically highly conserved — alterations could mean loss of binding function, stymying organism survival.

Here’s an example for the same protein we looked at earlier, 1A03:

    Last position-specific scoring matrix computed, weighted observed percentages rounded down, information per position, and relative weight of gapless real matches to pseudocounts                A   R   N   D   C   Q   E   G   H   I   L   K   M   F   P   S   T   W   Y   V   A   R   N   D   C   Q   E   G   H   I   L   K   M   F   P   S   T   W   Y   V        1 M    -3  -4  -4  -5  -4  -3  -4  -5  -4  -1   0  -4  10  -2  -5  -4  -3  -4  -3  -1    0   0   0   0   0   0   0   0   0   0   0   0 100   0   0   0   0   0   0   0  2.17 0.45        2 A     1  -3   0  -2   1  -2  -1   1  -3  -3  -2  -2  -3  -4   6   0  -1  -4  -4  -3   14   0   5   0   4   0   3  12   0   0   5   0   0   0  48   7   2   0   0   0  0.92 0.21        3 S    -1   0  -1  -2   1  -2  -2  -2  -2  -1  -3  -1  -2  -3  -2   4   4  -4  -3  -2    2   6   0   1   3   0   1   1   0   4   1   2   0   0   0  40  37   0   0   1  0.61 0.25        4 P    -2  -2  -2   3  -5   4   3  -3  -2  -4  -3   1  -3  -5   4  -1  -2  -5  -4  -4    0   0   0  15   0  23  22   0   0   0   3   9   0   0  25   1   0   0   0   0  0.80 0.28                                                ....       86 E    -2  -1   0   3  -4   0   4  -3   2  -3  -2   3  -3  -4   0  -2  -2  -4  -1  -3    0   0   3  20   0   0  33   0   5   1   5  26   0   0   5   0   0   0   2   0  0.55 0.20       87 A    -1  -1   0   0  -3   2   0  -3   2  -1  -2   2  -2   3  -3   0  -1  -1   3  -1    4   0   3   6   0  11   5   0   6   4   0  17   0  14   0   9   0   0  15   6  0.21 0.12       88 L    -2  -3  -3  -4  -3  -3  -4  -4  -3   1   3  -3   3   5  -4   1  -1  -1   2  -1    0   0   0   0   0   0   0   0   0   7  28   0   9  34   0  14   2   0   6   0  0.54 0.19       89 K    -3   0  -1   0  -5   2   2  -3   9  -5  -4   2  -3  -3  -3  -2  -3  -4   0  -4    0   2   0   5   0   7  11   0  62   0   0  12   0   0   0   0   0   0   0   0  1.18 0.25       90 G     0  -2   3  -1  -3  -2  -2   5  -2  -4  -4  -2  -3  -4  -3   0  -2  -3  -3  -3    6   0  23   0   0   0   0  72   0   0   0   0   0   0   0   0   0   0   0   0  0.87 0.08                              K         Lambda    Standard Ungapped    0.1364     0.3155    Standard Gapped      0.0519     0.2670    PSI Ungapped         0.1697     0.3169    PSI Gapped           0.0519     0.2670

NCBI provides PSI-BLAST as a command-line tool, so we’ll write some code that invokes it for all of our sequences.

5. Structural information using BioPython

Now that we’ve extracted our information from the sequences, let’s take a closer look at the 3D models.

Secondary structure codes, half-sphere exposure, and from DSSP

The Dictionary of Secondary Structure in Proteins (DSSP) determines the most likely secondary structure assignment of a residue given a 3D structure. While the original paper uses SPIDER2 to predict this, we can use the PDB itself.

# helpers.py    def hse_and_dssp(pdb_file: str):    '''    Get the HSE and DSSP codes of a PDB's residues.    '''    with warnings.catch_warnings():        warnings.simplefilter(action='ignore', category=FutureWarning)                    # Initialize PDBParser        pdb_parser = PDBParser()        structure = pdb_parser.get_structure('structure', pdb_file)                    hse = HSExposure.HSExposureCA(structure)        dssp = DSSP(structure[0], pdb_file)                    # Half-sphere exposure values        hse_up, hse_down, pseudo_angle = zip(*[(res[1][0], res[1][1], res[1][2]) for res in hse.property_list])                   # Extract DSSP values into separate lists        ss, asa, phi, psi = zip(*[(res[2], res[3], res[4], res[5]) for res in dssp.property_list])                    print(f'\\\\rData acquired for {pdb_file}')        return hse_up, hse_down, pseudo_angle, ss, asa, phi, psi    def safe_hse_and_dssp(pdb_file):    '''    Helper function to handle errors and apply `hse_and_dssp`.    '''            # Empty list to store error-producing filenames    error_files = []    try:        return hse_and_dssp(pdb_file)    except Exception as e:        print(f'Error processing file: {pdb_file} - {e}')        error_files.append(pdb_file)        return [None] * 7  # Return a list of None values to match the expected output structure

Let’s break down each of these features:

# gendata.py    print('Adding HSE, ASA, and DSSP codes...')    ## Add HSE, ASA, DSSP codes, etc.peptide_list[['Peptide HSE Up', 'Peptide HSE Down', 'Peptide Pseudo Angles', 'Peptide SS', 'Peptide ASA', 'Peptide Phi', 'Peptide Psi']] = peptide_list['Peptide Path'].apply(lambda x: pd.Series(safe_hse_and_dssp(x)))peptide_list[['Protein HSE Up', 'Protein HSE Down', 'Protein Pseudo Angles', 'Protein SS', 'Protein ASA', 'Protein Phi', 'Protein Psi']] = peptide_list['Protein Path'].apply(lambda x: pd.Series(safe_hse_and_dssp(x)))

Before we wrap up, there are a few more housekeeping steps we need to take care of:

    # One-hot encoding function    def one_hot_encode_array(ss_array):        ss_array = literal_eval(str(ss_array))        length = len(ss_array)        encoding = {code: [0] * length for code in dssp_codes}        for i, code in enumerate(ss_array):            encoding[code][i] = 1        return encoding        def one_hot_encode_row(row):        pep_encoded = one_hot_encode_array(row['Peptide SS'])        prot_encoded = one_hot_encode_array(row['Protein SS'])        new_data = {}        for code in dssp_codes:            new_data[f'Peptide SS {code}'] = pep_encoded[code]            new_data[f'Protein SS {code}'] = prot_encoded[code]        return pd.Series(new_data)    def extend_hse(hse: str) -> str:        \"\"\"        Extends a HSE to the full length of the peptide.        \"\"\"        hse = list(literal_eval(str(hse)))        hse = [hse[0]] + hse + [hse[-1]]                return hse    ## One-hot encode SS codes    ss_columns = peptide_list.apply(one_hot_encode_row, axis=1)    peptide_list = pd.concat([peptide_list, ss_columns], axis=1)        print('\\\\033[1mHSE, ASA, and DSSP codes added and encoded.\\\\033[0m')        ## Extend HSE and Pseudo Angles to match peptides length. Achieved by duplicating terminal values.    peptide_list['Protein HSE Up'] = peptide_list['Protein HSE Up'].apply(extend_hse)    peptide_list['Protein HSE Down'] = peptide_list['Protein HSE Down'].apply(extend_hse)    peptide_list['Protein Pseudo Angles'] = peptide_list['Protein Pseudo Angles'].apply(extend_hse)        peptide_list['Peptide HSE Up'] = peptide_list['Peptide HSE Up'].apply(extend_hse)    peptide_list['Peptide HSE Down'] = peptide_list['Peptide HSE Down'].apply(extend_hse)    peptide_list['Peptide Pseudo Angles'] = peptide_list['Peptide Pseudo Angles'].apply(extend_hse)        print('\\\\033[1mHSE data extended.\\\\033[0m')

Finally, we’ll reduce the table so proteins and peptides are no longer sharing rows before converting the whole dataset into a tabular array matching our initial specifications:

    # gendata.py        ## Reduce to one peptide/protein per row    peptide_cols = [col for col in peptide_list.columns if 'Peptide' in col] # Columns we want to keep in pep_data    peptide_cols.remove('Peptide Length')    protein_cols = [col for col in peptide_list.columns if 'Protein' in col] # Columns we want to keep in pro_data        pep_data = peptide_list[peptide_cols] # Select the peptide data    pro_data = peptide_list[protein_cols] # Select the protein data        # Combine the two dataframes    combined_data = pd.DataFrame(np.vstack([pep_data, pro_data.values]))    combined_data.columns = pro_data.columns        # Replace the original peptide_list with the combined data    peptide_list = combined_data

Next, we’ll define our table-making function:

    # helpers.py        def make_tabular_dataset(row: pd.Series) -> pd.DataFrame:         '''        `make_tabular_dataset` is designed to be applied to a row of the input DataFrame        to create a feature array.        '''        feature_dict = row.to_dict()                # get sequence and PSSM        sequence = feature_dict[f'Protein Sequence']        pssm = feature_dict[f'Protein PSSM']                # remove the first row of the PSSM, which is the sequence        pssm = pssm.iloc[1:]        pssm = pssm.values                # get the binding indices list        binding_indices_dummy = [0] * len(sequence)        binding_indices = feature_dict[f'Protein Binding Indices']        for i in range(len(sequence)):            if i+1 in literal_eval(binding_indices):                binding_indices_dummy[i] = 1                # add the sequence as the first element of the array        arr = []        arr.append([char for char in sequence])                 # remove these columns from the dictionary; all are either unneeded or have been processed        remove = ['Unnamed: 0', 'PDB ID',                  'Number of Atoms in Protein', 'Protein Chain ID', 'Protein Path',                  'Number of Atom Contacts', 'Resolution',                  'Molecular Type', 'Protein Path', 'Protein Sequence',                  'Protein Binding Indices', 'Protein SS', 'Protein PSSM']                 # many of the features are stored as strings, so we need to convert them to lists        use_feature_dict = {}        for k, v in feature_dict.items():            if k not in remove:                try:                    use_feature_dict[k] = literal_eval(str(v))                except (ValueError, SyntaxError) as e:                    print(f\"Error parsing key '{k}' with value '{v}': {e}\")                            # stack all the features        for _, value in use_feature_dict.items():            value_array = [residue_value for residue_value in value]            arr.append(value_array) # add the feature to the array        for aa_row in pssm:            arr.append(aa_row)        arr.append(binding_indices_dummy)                # transpose the array and add the column names        arr = pd.DataFrame(arr).T        columns = ['AA'] + list(use_feature_dict.keys()) + list('ARNDCQEGHILKMFPSTWYV') + ['Binding Indices']                if len(arr.columns) == len(columns):            arr.columns = columns        else:            print(\"Column length mismatch\")            print(\"Arr shape:\", arr.shape)            print(\"Columns length:\", len(columns))                return pd.DataFrame(arr)
    # gendata.py        ## Create tabular dataset, 1 row per residue    list_of_feature_arrays = []    for i, row in peptide_list.iterrows():     \tarr = make_tabular_dataset(row)    \tlist_of_feature_arrays.append(arr)    \tprint(f'\\\\r{i+1}/{peptide_list.shape[0]}', end='')        print('\\\\033[1m\\\\nConverted.\\\\033[0m')        export = pd.concat(list_of_feature_arrays)    export = export.dropna()    export = export.reset_index(drop=True)    export.to_csv(peppi_data_csv, index=False)

But let’s not forget — we also wanted to generate an image dataset. Here’s how we do that with our tabular dataset.

Optional: Generating an image dataset

We need to write functions that:

    # helpers.py        def window_maker(sequence_array: pd.DataFrame) -> List[pd.DataFrame]:        '''        Takes a sequence array and returns a list windowed arrays by sliding a window over the input feature array.         The window is centered on the residue of interest.         '''                k = 7 # window size        windows = []        sequence_array = sequence_array.T.reset_index(drop=True)        sequence_length = sequence_array.shape[1]                        for i in range(sequence_length):                        # N-terminal case            if i < 3:                                right = sequence_array.iloc[:, 0:i+4]                                if i == 0:                    left = right.iloc[:, :-4:-1]                elif i == 1:                    left = right.iloc[:, 1::-1]                elif i == 2:                    left = right.iloc[:, 1]                                window = pd.concat([left, right], axis = 1)                        # C-terminal case            elif i >= sequence_length - 3:                                left = sequence_array.iloc[:, i-3:]                                if i == sequence_length - 3:                    right = sequence_array.iloc[:, i+1]                elif i == sequence_length - 2:                    right = sequence_array.iloc[:, i-2:i].iloc[:, ::-1]                elif i == sequence_length - 1:                    right = sequence_array.iloc[:, i-3:i].iloc[:, ::-1]                                window = pd.concat([left, right], axis = 1)                        # Standard case            else:                window = sequence_array.iloc[:, i-3:i+4]                        windows.append(window)        return windows        def create_images(window: pd.DataFrame, name: str):        '''        Helper function that accepts a window DataFrame and converts it         into a CNN-friendly image.        '''        window = window.T.reset_index(drop=True) # transpose the array and reset the index        window = window.apply(pd.to_numeric, errors='coerce') # convert to numeric        window = window.to_numpy() # convert to numpy array        window_normalized = window * 255 # scale to 0-255                    window_uint8 = window_normalized.astype('uint8')        img = Image.fromarray(window_uint8)        img.save(name)        print(f'\\\\rCreated image: {name}.', end='')            def process_images(list_of_feature_arrays: List[pd.DataFrame], binding_path: str, nonbinding_path: str):        '''        Takes a list sequence feature arrays and uses create_images to turn valid ones into images        in the appropriate folder.        '''        os.makedirs(binding_path, exist_ok=True)        os.makedirs(nonbinding_path, exist_ok=True)        name_index = 0                for arr in list_of_feature_arrays:            residues = arr.pop('AA')                        binding_indices = arr.pop('Binding Indices')            binding_indices = binding_indices.to_list()                        windows = window_maker(arr)            for i, window in enumerate(windows):                if not window.isna().any().any():                    name_index += 1                    folder = binding_path if binding_indices[i] == 1 else nonbinding_path                    name = f'{folder}/{name_index}.jpg'                    create_images(window, name)                else:                    continue        print('\\\\n')

There’s a lot to unpack here, so let’s take it function by function:

# gendata.py    ## Optional step to create images:if args.images:    print('\\\\033[1m\\\\nNow creating images...\\\\033[0m')    process_images(            list_of_feature_arrays,             binding_path=args.binding_path,             nonbinding_path=args.nonbinding_path    )

And there we have it!

Behold — the PepBDB-ML database.

\"Visualization

For the data-curious among us — don’t worry. Here are some visualizations to explore the dataset now that we have it available:

The Results: Visualizing the PepBDB-ML database

Amino acid distributions

![Figure 1. Bar chart of amino acid counts in PepBDB-ML. By Author](https://cdn-images-1.medium.com/max/2000/1*ch6d7U1KXSY5sZGQWkqqbw.png)

Figure 1. Bar chart of amino acid counts in PepBDB-ML.

There isn’t too much to note here, apart from the curious observation that Leucine outnumbers every other amino acid in the database.

\"\"

![Figure 2. Bar chart of amino acid distributions in binding (left) and non-binding residues (right). By Author](https://cdn-images-1.medium.com/max/2000/1*rNMg8NvXDalEcU-31bD6MA.png)

Figure 2. Bar chart of amino acid distributions in binding (left) and non-binding residues (right).

Splitting our amino acid counts by binding vs. nonbinding residues shows us the relative proportion of AAs differs ever so slightly. Here’s a view of both in the same plot:

![Figure 3. Bar chart of amino acid percentages by class in PepBDB-ML. By Author](https://cdn-images-1.medium.com/max/2000/1*lpnFm06hwr6-cM1Tw5hpMA.png)

Figure 3. Bar chart of amino acid percentages by class in PepBDB-ML.

Takeaways:

Correlation mapping

![Figure 4. Correlation heatmap of features in PepBDB-ML. By Author](https://cdn-images-1.medium.com/max/2742/1*2KW9mkpGbjgxbcbEUBXT_w.png)

Figure 4. Correlation heatmap of features in PepBDB-ML.

A correlation heatmap helps us visualize correlations between variables in our dataset. The colours and labels tell us the strength and directions of those correlations.

Some takeaways:

Feature distributions

![Figure 5. Distributions of 41 features found in PepBDB-ML. By Author](https://cdn-images-1.medium.com/max/3002/1*rL-BeJZjiMjtAfnxRNKgFA.png)

Figure 5. Distributions of 41 features found in PepBDB-ML.

Some interesting callouts:

Now, let’s compare distributions between binding and non-binding residues:

![Figure 6. Distributions of 41 features found in PepBDB-ML by class. By Author](https://cdn-images-1.medium.com/max/3020/1*fPzkkcQQRdKFv5EBQucAgg.png)

Figure 6. Distributions of 41 features found in PepBDB-ML by class.

The number of nonbinding residues far outnumbers binding residues, so it’s a little tricky to parse in this visualization — but feature distributions appear quite similar between samples.

Does this mean there isn’t a discernible difference between our samples? This could cause some issues if we plan on using this dataset for training classifiers. Let’s do some quick statistical analysis:

Statistical tests

The Discussion: Key takeaways and learnings

Developing this dataset was step 1 in trying to reimplement the code from the paper I mentioned at the start.

I (naively) assumed the data construction would be the easy part. In reality, the bulk of my time was spent wrangling the myriad of enrichment tools and databases, accounting for edge cases in all of them, and formatting.

But it paid off.

The CNN I trained on this data outperforms the original by 12 pts. on binding site detections (79%) — but I’ll save that for a future article.

Key takeaways:

You can download the full dataset at my GitHub or my website. Running the data generation scripts can take some time, so ensure you have enough computing power or access to an HPC.

Let me know if you have any feedback on improving — thanks for reading!

", "url": "https://murto.co/pepbdb-ml/", "image": "/img/seo/pepbdb-ml-cover.png", "tags": ["bioinformatics","ml"], "date_published": "2024-08-01T00:00:00+00:00", "date_modified": "2024-08-01T00:00:00+00:00", "author": { "name": "Murto Hilali", "url": "https://murto.co/", "avatar": "https://murto.co/img/author/website-headshot.jpeg" } }, { "id": "https://murto.co/hyperjudge/", "title": "Hyperjudge: Source-Checking Tool for Doc Validity", "summary": "A web app that checks the validity of linked sources in documents and webpages.", "content_text": " Overview The Plan, Problem, \\& Solution Lessons \\& Takeaways RoadmapOverviewEarlier this year, I did editing work for my friend Tameem’s marketing agency, TalktheTalk Creative. I had a few editing tasks associated with this: Checking spelling and grammar Splitting up extensive paragraphs Ensuring factual correctnessPart of that last bullet inlcudes checking the sources an articles links to and ensuring they’re: From reputable sources. The fact/stat/figure linked was congruent with the source.Bullet #2 is a bit trickier to automate (I’ll probably need to use an LLM) but #1 is more straightforward.The Plan, Problem, & SolutionIn essence, all I needed to do was: Extract all the hyperlinks and associated text from the body of the page. Get the authority score from the domain via Moz API. Return the DA scores and average graphically.However, there was one problem — my ultimate goal is to ship this as a Chrome extension. That means it needs to be written in JavaScript. One problem: the only JavaScript I know is console.log('Hello, World!).Luckily, I have access to GPT-4o. Combining a high-fidelity mockup:Along with with lots of debugging and iced coffee led to the birth of Hyperjudge:I was able to build this under a day — I’m sure there are some cracked engineers who could do it an hour or two, but as someone with close to 0 experience with anything other than Python, I’m pretty proud of myself!Lessons & Takeaways LLMs are superpowers — but only for those who know how to use them. Trying to build this project without GPT-4 would probably have taken me days, even weeks. Because of OpenAI (thank you Sam) I could shrink that down to < 24 hours. However, debugging would have been a nightmare if I had absolutely 0 familiarity with programming. LLMs are never going to give you perfect, ship-ready code out of the box, so knowing fundamentals is crucial for building things that work. It’s much easier to build side projects as gifts. One of my goals with Hyperjudge was to help automate one of the tasks I used to execute for Tameem — since I had a user in mind, it was a lot easier to stay motivated and on track.Roadmap A Chrome extension that automatically checks the source validity of a page or Google Doc. Add a feature that checks to ensure the hyperlinked text aligns with the source. (For instance, the source says, ‘2 out of 3 people love Murto’s blog posts on Medium’ and the hyperlink says ‘67% of readers enjoy Murto’s blog posts’).", "content_html": "

\"Hyperjudge

Overview

Earlier this year, I did editing work for my friend Tameem’s marketing agency, TalktheTalk Creative. I had a few editing tasks associated with this:

Part of that last bullet inlcudes checking the sources an articles links to and ensuring they’re:

  1. From reputable sources.
  2. The fact/stat/figure linked was congruent with the source.

Bullet #2 is a bit trickier to automate (I’ll probably need to use an LLM) but #1 is more straightforward.

The Plan, Problem, & Solution

In essence, all I needed to do was:

However, there was one problem — my ultimate goal is to ship this as a Chrome extension. That means it needs to be written in JavaScript. One problem: the only JavaScript I know is console.log('Hello, World!).

Luckily, I have access to GPT-4o. Combining a high-fidelity mockup:

\"\"

Along with with lots of debugging and iced coffee led to the birth of Hyperjudge:

\"Hyperjudge

I was able to build this under a day — I’m sure there are some cracked engineers who could do it an hour or two, but as someone with close to 0 experience with anything other than Python, I’m pretty proud of myself!

Lessons & Takeaways

Roadmap

  1. A Chrome extension that automatically checks the source validity of a page or Google Doc.
  2. Add a feature that checks to ensure the hyperlinked text aligns with the source. (For instance, the source says, ‘2 out of 3 people love Murto’s blog posts on Medium’ and the hyperlink says ‘67% of readers enjoy Murto’s blog posts’).
", "url": "https://murto.co/hyperjudge/", "image": "/img/seo/hyperjudge-cover.png", "tags": ["seo","writing","tools"], "date_published": "2024-05-15T00:00:00+00:00", "date_modified": "2024-05-15T00:00:00+00:00", "author": { "name": "Murto Hilali", "url": "https://murto.co/", "avatar": "https://murto.co/img/author/website-headshot.jpeg" } }, { "id": "https://murto.co/xgboost-ppi/", "title": "Predicting the Effect of Mutations on PPIs Using AlphaFold", "summary": "How I used AlphaFold-Multimer, XGBoost, and 47,000 SLURM jobs to predict PPI outcomes with 91% AUC", "content_text": "The human interactome, encompassing all protein-protein interactions, may involve up to 600,000 interactions.Predicting how a disease-causing mutation affects the interactome might seem like a Herculean task—but it’s not as impossible as you might expect, especially when you give a University of Waterloo co-op student free access to a beefy GPU cluster, world-class mentorship, and free agency to pursue any approach.I used the machine learning framework XGBoost, cutting-edge deep learning software AlphaFold-Multimer (AF-M), and over 47,000 SLURM jobs to build a multi-classifier model that predicts the effects of missense mutations on PPIs with a 91% AUC.Multi-class ROC curve and AUC (By Author)In this article, I’ll walk through: The Background: The research question and why we chose it. Data Acquisition & Processing: How and why we acquired the protein data we did, along with preprocessing steps. The Machine Learning Model: The approach for selecting a model and how I implemented it for protein data. Results, Model Accuracy, & Feature Importances: Understanding the model’s results and analyzing the important features, class by class. Case Studies on ASD-Related Proteins: A deeper dive into two specific model results, focusing on autism spectrum disorder-related proteins. Final Thoughts and Lessons: Conclusions, key takeaways, and future directions for this project.If you’re involved in bioinformatics or molecular biology and want to start incorporating machine learning into your research—this article is for you.Let’s jump into it!BackgroundIn this section, I’ll cover: The research question under investigation. The rationale for choosing this question.Not so hot take: protein-protein interactions are important.I might be biased because I studied them for eight months at The Center for Applied Genomics at SickKids—but I’ve got good reason to think so: PPIs are involved in countless metabolic processes. Mutations that affect PPIs are overrepresented in diseases (Cheng et al., 2021).So, if we can understand how and why a given mutation impacts PPIs, there’s a lot we can do: Get a fuller picture of disease mechanisms. Design better drugs to treat those diseases.With DeepMind’s release of AlphaFold-Multimer (AF-M), a version of AlphaFold trained with protein complexes in mind, came an opportunity and a question: Can AF-M capture the effect of missense variants on PPIs?You might be wondering—why missense variants specifically? As opposed to other kinds of mutations?Well… AF-M can’t predict macro-level structural changes from point mutations. However, in silico changes at the interface of a PPI can have measurable changes (observed through a program like ChimeraX).It’s a research question that lets us take advantage of current technology.Now—how do we go about answering this question? I’ll answer that in the next section.Methodology & ApproachIn this section, I’ll cover: The central idea behind how we frame this problem. A broad-stroke overview of what we’re trying to learn.3D structures — like the kind AF-M generates — have exclusive observable properties you can’t get anywhere else.Most of these are structural features like: Protein-protein interface area Shape complementarity Docking scoreThis is exciting data we can only observe from PDB structures. Given the power of a 3D structure, we arrived at the following approach: Take a wildtype protein complex (just a regular old homo- or hetero-dimer). Now, take the same complex, except this time with a missense variant in one of the proteins. Effectively, a ‘missense complex.’If we can detect the difference between the structural features of these complexes and relate it to the specific missense variant, we can find a relationship:Impact on PPI ~ (structural features).In other words, impact on PPI as a function of structural features.Illustration of wildtype and missense complex comparisonBut there’s another opportunity here: what if I enriched that structural data with variant annotations, too? Examples include: Pathogenicity predictions (CADD, REVEL, AlphaMissense) Pathogenicity annotations from ClinVar Allele frequencyWe’d have many more features to parse a relationship from, developing a more comprehensive function:Impact on PPI ~ f(x₁, x₂, …, xn)During my data collection process, I was able to add 40+ more non-structural features on top of the structural ones. This made for excellent training data for an XGBoost classifier model that could predict the Impact on PPI of a given variant out of these four classes: Increase in interaction strength. (Binding improves between the proteins) Decrease in interaction strength. (Binding weakens between the proteins). Disruption of interaction. (The proteins no longer interact with each other). No effect on interaction. (The variant does not alter the quality of interaction).You’re probably wondering—how did I get all this data matching missense variants to PPI effects? And why did I choose XGBoost as my classifier model?I’ll explain in the following sections.Data Acquisition & ProcessingIn this section, I’ll cover: Where I obtained my data (IntAct mutations database). How I preprocessed and engineered features from that data.IntAct Mutations DatabaseI gathered my training data from the IntAct Mutations database (available under license CC0). It’s a massive annotated record with thousands of missense variants involved in binary protein interactions.Every variant is annotated with its effect on PPI, encoded under the ‘Feature type’ column: increase, decrease, no effect, etc.:Data encoding in IntAct Mutations databaseHowever, you’ll notice this dataset does NOT include structural feature information or other variant annotations. That’s where I (by which I mean AF-M) came in.The database contains ~12,000 eligible data points. To save compute and ensure balanced data classes, I randomly subsampled variants until I had ~1,000 for each class. Curious about how random subsampling works? Random subsampling is when I randomly pull datapoints from a given class until I have enough. This is one approach to ensuring your classes are weighted equally in your training data set so your classifier doesn’t bias towards any single one.Of course, you could also simply assign different weights to each class instead, but random subsampling means I have fewer protein complexes to fold—saving on compute.Overview of data aggregation pipelineI then wrote a script that generated FASTA files for the wildtype and missense complexes before feeding them into AF-M to produce PDB structures. Due to time constraints, the final dataset had about 250 structures per class.Creating a wildtype and missense complex Wondering how I went from UniProt IDs to PDB structures?This was a relatively simple Bash script that would pull UniProt IDs from the Intact Mutations file and download the appropriate FASTAs via CURL request. We create a wildtype FASTA and then ‘mutate’ a copy to make a missense version. These are inputs for AF-M.This leaves us with two PDB versions of each protein complex: a wildtype and a missense variant (see image above). Multiply that across ~a thousand binary interactions and 47,000 SLURM jobs, and we have ~20TB worth of data to work with.Feature Extraction & EngineeringAll I had left to do was extract structural and variant feature information:Overview of data aggregation pipeline, final stepsThe AF-M pipeline I used, AlphaPulldown, simulates a pulldown assay by parallelizing the ‘folding’ of several protein complexes. It also includes a feature extraction pipeline that generates several key structural metrics: Interface area (Ų) Number of interface residues Shape complementarity Percentage of polar/hydrophobic/charges residues at the interface Docking score Protein-Interface scoreAnd beyond. I also added a few features of my own, with annotations from Variant Effect Predictor by Ensembl, available under license Apache 2.0:Pathogenicity predictionsTwo examples of pathogenicity predictions I used: AlphaMissense annotations. These are pathogenicity predictions for every possible missense mutation in the human proteome. REVEL pathogenicity predictions. REVEL is a pipeline that produces an average score of several other pathogenicity prediction tools.Why (and how) I included them: If pathogenic missense variants are enriched at PPIs, then the AlphaMissense and REVEL scores will likely be strong predictors of PPI disruption. I determined these values by converting the IntAct mutations into a VEP-readable format.gnomAD frequenciesgnomAD (the Genome Aggregation Database from the Broad Institute) contains population allele frequencies for several different groups and variants.Why (and how) I included frequencies: Variant frequency data could also help us learn whether common or rare variants are more prevalent in PPI-disrupting pathologies. I found these values by converting the IntAct mutations into a VEP-readable format.Relative ratiosI engineered simple ratio features, like the interface area / total surface area, the differences in feature x from the wildtype version, etc.Why I included relative ratios: While an unsupervised algorithm could make these kinds of calculations without help, a supervised model like XGBoost benefits from these ratios. Intuitively, we could also predict that if the ratio of interface area to total surface area falls, we’re observing a weakened interaction. (This is just one example).Free energyUsing EvoEF2 (available under the MIT license), I obtained thermodynamic data on protein complexes, comparing 𝚫𝚫G (the difference in Gibbs energy of the folded and unfolded protein states) of the wildtype and mutant variants.Why free energy: We’d expect higher free energy predictions from a complex with PPI-disturbing mutations, suggesting a more unstable interaction.By the end of it all, I had a tabular dataset that looked a bit like this:One thing about bioinformatics tools, however—they don’t always work. Metrics for a given variant will often be empty or missing. For example, not every variant in IntAct has an associated gnomAD frequency. Wondering why can missing values make ML challenging?It probably makes intuitive sense as to why missing values are problematic in any machine learning task. Given an incomplete picture of the dataset, it can be hard to trust the output of your model. Luckily, there are methods for dealing with our trust issues.Missing values are pretty annoying whenever you do machine learning, so I had to pick a model with the right chops. Luckily, XGBoost had what it took—let me explain how.The Machine Learning ModelIn this section, I’ll break down: XGBoost, and why I chose it. Hyperparameter tuning & cross-validation.XGBoostXGBoost is a gradient-boosted decision tree model.Decision trees are graphs that split data at threshold-based nodes. Here’s a simple model of deciding whether or not to play outside today:Decision tree diagramChaining several of these trees together in sequence allows each one to correct on the error of the last one, building a much more robust model. This is what makes it ‘gradient-boosted.’XGBoost is a lightweight, fast-training model that has several advantages for this particular task: It doesn’t need normalized data. This is great for several reasons. For one, it means less work for me. More importantly, however, it makes it much more straightforward to interpret how the model arrived at a conclusion. It can handle multiple data types. This includes categorical and nominal data—this is a more experimental feature, however, and I simply used encoded dummy variables myself for the categorical data. It’s a supervised model. Artificial intelligence is doing incredible things for science, but it’s easy to get lost in the code. Ultimately, the goal here is to learn something about biology. Since supervised models let us pop the hood behind classifications, we can investigate the biological mechanism behind a PPI outcome in more detail. It has robust handling for missing data. Remember how I mentioned not all variants have full annotations? They might be missing frequency information, for example. Or maybe EvoEF2 bugged out, and I need thermodynamic information. XGBoost has methods to ensure this doesn’t shut down training altogether. How does XGBoost handle missing data?The model treats missing data as essentially their own kind of value. At a splitting node, the algorithm determine which direction (left/right) leads to the highest gain (best reduction of the loss function) and encodes that decision into its training. When it encounters a similar missing value in the test set, it’ll make the same decision.Since the dataset had reasonably sized dimensions and XGBoost trains so quickly, I decided to include a hyperparameter tuning step using k-fold cross-validation:Hyperparameter tuning & k-fold cross-validationAny machine learning model will have internal parameters—the weights and biases it ascribes to its internal classifications.However, ML models also have hyperparameters, which are higher-level structural settings for the overall model. These parameters are set before training and can significantly impact accuracy outcomes.Here are the ones I focused on: Max_depth: max depth of a tree Learning_rate: contribution of each tree to the final result N_estimators: number of trees Subsample: training set fraction sampled for each treeWe determine the optimal hyperparameters by testing several combinations and selecting the one with the best results—in other words, by tuning.To add even more robustness to the model training, I implemented a k-fold cross-validation step. What is that exactly?Let’s break it down: In ML, we typically split the dataset into a training and test set. Sometimes, we risk overfitting the model to the training data, making our model less useful. To combat this, we can split our dataset into a number (k) of segments. Then, for k different models, we select one segment as our test set and combine the rest as our training data. We repeat this for k segments, shuffling our data so we never overfit to just one training/testing split.By MBanuelos22 from Wikimedia Commons, CC BY-SA 4.0We do this for every hyperparameter combination, ensuring we get the highest accuracy model possible.Now that we’ve got a model trained and tested, how good is it? What can it teach us about biology? We’ll cover that next:Results, Model Accuracy & Feature ImportancesIn this section, I’ll cover: How accurate the model is, and what those accuracy metrics mean. Class by class, which features are the strongest predictors.Confusion matrix & ROC curveTo assess the quality of a multi-classifier, you’ll typically use a confusion matrix:Figure 1. Confusion matrixThe confusion matrix visualizes the test set. Along the bottom, we have what the model predicted for each data point (a protein complex), and on the y-axis, we have the actual value for each data point.The lighter the colour of the diagonal, the more accurate the model. For instance, the model correctly predicted 47 of the 54 class 3 protein complexes (see the bottom row). We see a similar level of accuracy in class 2, with 39/46 complexes correctly classified. In the upper left corner, the model had difficulty distinguishing between classes 0 and 1 (mutations decreasing and disrupting interactions, respectively). Intuitively, this makes sense—they have similar effects.The confusion matrix is just one way of assessing the model’s accuracy. We can also use an ROC curve:Figure 2. Multi-class ROC curve and AUCA receiver operating characteristic (ROC) curve plots FPR vs. TPR: True positive rate (TPR): the ratio of true positives to actual positives [TP/(TP+FN)]. False positive rate (FPR): the ratio of false positives to actual negatives [FP/(FP+TN)].The points along the curve indicate different threshold settings. The threshold refers to the cutoff point where we distinguish between a positive and negative case.In a multi-class ROC curve, the positive case is a given class, and the negative case is all the others (one-vs-all). The diagonal (dashed line) refers to random chance, where TPR = FPR at all thresholds. One way to think of accuracy is that the further we are from this line, the better the model. At the left bottom of the graph, a high threshold can lead to few false positives (low FPR) but also many missed true positives (low TPR). As you move right, the threshold decreases allowing more true positives (high TPR) but also letting in more false positives (high FPR).The ideal classifier has a bowed-out shape, where we have a high TPR at low FPR rates, even at high thresholds. We see this for most of our curves. TPR = ∑c​TPc / ∑c​(TPc​+FNc​) FPR = ∑c​FPc / ∑c​(FPc​+TNc​)​​​​Microaveraging the curves involved summing each class’s TPs, FNs, and TNs (seen above).This gives us an overall picture of our model’s performance. To standardize our comparison, we calculate the area under the curve. Closer to 1 = more accurate. As seen in Figure 2, we achieve a micro average AUC of 0.91.You can also see that the curves for classes 0 and 1 reflect what we saw in the confusion matrix. Overall, we have a reasonably accurate model. But now comes the fun part—what can it tell us about biology?SHAP valuesUsing SHAP values can help us determine how much the features in a model contributed to each prediction. What are SHAP values?Shapley Additive exPlanations measure the marginal contributions of each feature to a class prediction. In short, they measure feature importance. SHAP values are useful because they work for any supervised machine learning task.They’re determined by: Taking the average prediction for a given class. Iteratively adding each feature in (in all possible orders) and evaluating the output. Calculating the marginal effect of a feature on the model output’s distance from average.SHAP values are biologically useful because they can tell us which structural features require more study, or which pathogenicity predictions are most accurate.Here are the top ten feature importances for the entire model:Figure 3. Feature importanceFrom my analysis, I saw some interesting results. For one, the impact of the resulting amino acid from the missense mutation being cysteine (C) was surprising and challenging to explain. Not so surprising was the importance of the AlphaMissense pathogenicity score—both AlphaPulldown and AlphaMissense (in one way or another) rely on the MSAs from AlphaFold. Interestingly, the magnitude of the difference in the percentage of charged residues at the interface between wildtype and missense complexes also plays an important role.Realistically, finding one or two features globally predictive of PPI effects is a fruitless exercise. We’re better off looking at feature importance class by class using SHAP values:Class 0: Decrease in interaction strengthOn the left, I have a feature importance plot showing the average impact of a feature on the model’s output. This graph is relatively simple to interpret; the bigger the bar, the greater the magnitude of impact.On the right, we have a beeswarm plot. These kinds of plots help us map two factors: Impact on outcome likelihood: Dots on the right of the center line mean those feature values have a positive effect on the class prediction (i.e., they make it more likely to be predicted). Likewise, dots to the left have a negative effect (they reduce the likelihood). Feature value: The colour of the dots is also essential. Blue dots represent a low feature value, while red dots represent a high feature value.We can combine these two factors to further our understanding of the feature effects. I’ll use the Resulting_sequence_A feature as an example. (Background: this is a dummy value, so values are either 0 or 1). Dots to the right of the center are red: when Resulting_sequence_A is high, it increases the likelihood of predicting class 0. Dots to the left of the center are blue: when Resulting_sequence_A is low, it decreases the likelihood of predicting class 0.In natural language, it seems alanine as a missense residue has a strong effect on decreasing interaction strength. This tracks intuitively—alanine is a small amino acid with only a methyl group for a side chain.This limits the kind of interactions it can participate in. So, if it replaces a crucial disulfide bride or prevents a charged interaction, it makes sense that it would increase the likelihood of decreased interaction strength. Combining wet and computational biology for a better model.While there is a biological case to be made that this finding reflects reality, it’s also true that scientists often conduct alanine-scanning studies where residues are sequentially replaced in a protein sequence to determine which ones are important in folding, interaction, etc. It’s possible the results from these studies are overrepresented in the IntAct database, biasing the model to think this feature is more important than it is.Interestingly, it was crucial I have background in ‘wet’ biology to understand issues like this. Otherwise, I may not have even realized this was a potential problem.We see some unexpected results with the shape complementarity (sc) feature. sc measures how well the two proteins fit into each other; we’d expect that when sc is low, it increases the likelihood of decreased interaction strength between two proteins. However, this is not the case: higher sc seems to decrease interaction strength, and vice versa.In the following sections, I’ll only call out a few features I found interesting:Class 1: Disruption of interaction SIFT is a normalized variant effect prediction score between 0 and 1. Low scores (0–0.5) are predicted to affect protein activity. We see the expected results reflected in the beeswarm plot, adding to SIFT’s strength as a tool. Total_free_energy_diff_from_wt is an engineered feature that measures the difference in the EvoEF2 free energy between a missense complex and its wildtype equivalent. A significant difference (high value) suggests a more unstable missense complex (and v.v.), which appears to be reflected in the plot.Class 2: Increase in interaction strength Resulting_sequence_A: For whatever reason, missense mutations resulting in a cysteine residue have a surprisingly strong, positive effect on the likelihood of increasing interaction strength. Cysteine can form disulfide bridges across dimers, one of the stronger bonds. While this may lead to enhanced interactions, I wonder why this is the most impactful feature? iptm & iptm_ptm: These values measure AlphaFold’s confidence in the models. It seems to track that if two proteins ‘make sense’ in a complex (illustrated with a high iptm_ptm), they would have strong interactions. In the reverse case, if AlphaFold has low confidence in the structure, it could mean these proteins are unnatural in complexity, thus reducing the likelihood of class 2 prediction. However, it could also simply mean the 3D model is inaccurate and should be disregarded—I cannot make any solid conclusions.Class 3: No effect on interaction AM_PATHOGENICITY: This is a pathogenicity score from AlphaMissense based on conservation scores for every residue in the human proteome. It seems to be a relatively strong predictor of non-pathogenic mutations, with low pathogenicity scores associated with an increase in the likelihood of class 3 prediction. pDockQ: A docking metric that measures (like sc) how well two proteins complex together in 3D space. A good docking score, as expected, seems to be a strong predictor of a non-perturbational variant. gnomAD_exomes_AF: The variant’s exome allele frequency from gnomAD suggests low-frequency variants are associated with unperturbed protein complexes. However, we’d have to see the range of frequencies to determine if we’re looking at rare or common variants (These classifications require frequencies between specific thresholds).All in all, some interesting results. Now, let’s apply the model to some novel protein complexes to see if we can learn anything new:Case Studies: ASD-Related Protein ComplexesIn this section, I’ll: Look at two specific examples of protein complexes analyzed with the model. Provide a literature-based (potential) explanation for both.This is all fun and interesting, but what’s the point of machine learning in molecular biology if we haven’t learned anything new?So, for the application part of this project, I used the model to analyze a few coding missense variants from the MSSNG database, a repository of whole genome sequence and annotation data designed to study autism spectrum disorder (ASD). (The Scherer Lab specializes in studying the genomics of ASD).MSSNG is a vast body of data, so I refined my analysis to only a subset that met the following criteria: Coding missense variants (restricts for affected proteins). Ambiguous pathogenicity predictions from REVEL and AlphaMissense (so the model could help generate some new information). Have known direct interactions according to the IntAct database (the only type of interaction the model is trained on). Meet a PAE cutoff of 15 Å when folded using AlphaPulldown (restricts for high confidence in the relative positions of protein domains).Here are two of the standout variants from this subset: Androgen receptor & tyrosine protein kinase (AR:YES1.p.Q234R) Transcriptional coactivator & zinc finger protein (YAP1:GLIS3.p.G448D)The model predicts Class 1 (a disrupted interaction) for both variants. Let’s take a closer look:Androgen receptor & tyrosine protein kinase (AR:YES1.p.Q234R)Here are the facts: AR (the androgen receptor) is found in the heart, skeletal muscle, and the prostate (UniProt). Tyrosine phosphorylation of AR has been linked to tumour growth (Guo et al., 2006). Some evidence suggests an inverse comorbidity between ASD and prostate cancer (Fores-Martos et al., 2019).So here’s one potential story: A disrupted interaction between AR and YES1 reduces tyrosine phosphorylation, thus lowering cancer risk. Inverse ASD and prostate cancer comorbidities may explain this variant’s role in autism.Unfortunately, there are a few issues here: YES1 is only expressed at low levels in the prostate. Moreover, it’s the non-receptor tyrosine kinase. Plus, there is also evidence running counter to the ASD-prostate cancer comorbidity finding.The bottom line is that there are no conclusive results from the model. For your interest, here are the SHAP values that lead to this prediction. Note the comparatively high difference in interface solvation energy from the wildtype.Transcriptional coactivator & zinc finger protein (YAP1:GLIS3.p.G448D)Once again, here are the facts: YAP1 can coactivate/repress transcription and is found in the liver and the prostate (also at low levels in the brain). GLIS3 is also a transcriptional activator/repressor found in the kidneys, brain, and liver. Several GLIS3 binding sites are associated with neuropathologies, affecting nervous system development and synaptic transmission (Calderari et al., 2018). GLIS3 may also affect autophagy regulation (Calderari et al., 2018).One possible story here: Disrupted interaction between these two proteins may expedite the progression of neuropathologies that result in autism.However, gleaning a more specific mechanism from this research is a challenge.Once again, here are its SHAP values:So far we’ve trained a reasonably strong model, interpreted its results, and used it to learn more about biology. What are the takeaways? And what’s next?Final Thoughts and LessonsFuture DirectionsI spent about four months at TCAG working on this project, much of which was spent collecting training data. (Not to mention the four months I spent there previously getting accustomed to AF-M).Given more time, here are steps I’d take to expand the project: Gathering more data. With a limited dataset, there’s only so much faith we can put in the results from this model. More PDBs, ideally experimental ones, are step 0 to improve the model. Engineering new features. Taking a deeper molecular biology approach to feature engineering may increase the model’s accuracy. Use the model for pathogenicity prediction instead of mechanism prediction. Currently, the model predicts the impact on PPI (Feature type from the IntAct mutations database). An alternative approach would be using the enriched data for pathogenicity prediction, a la REVEL. Testing the model on more unexplained variants. As it stands currently, I was only able to test the model on a few variants of uncertain significance. Analyzing more PPIs could create further research threads to pull on.However, I’m quite proud of what I achieved and learned in this timeframe.Conclusions & Key TakeawaysI can’t imagine many students are given free rein over such an exciting research question, access to world-class mentors, and one of Canada’s largest compute clusters.But I did.So, I’m incredibly thankful to everyone involved (Dr. Brett Trost, Dr. Richard Wintle, and Dr. Steve Scherer, to name a few).Working in the Scherer Lab showed me research doesn’t have to be stale, slow, and tedious. With the right team and vision, it can be fast, dynamic, and cutting-edge.On a macro-scale, I’ve learned how crucial AI/ML will be to the future of computational biology. My initial approach to this project was creating a graph structure to observe and track the structural differences between wildtype and missense complexes. When I realized the magnitude of the statistics needed to determine the differences between the two, it dawned on me that ML could save the day here.What are your thoughts?I’m happy to get your feedback, questions, compliments, and curses. Thanks for reading!", "content_html": "

\"\"

The human interactome, encompassing all protein-protein interactions, may involve up to 600,000 interactions.

Predicting how a disease-causing mutation affects the interactome might seem like a Herculean task—but it’s not as impossible as you might expect, especially when you give a University of Waterloo co-op student free access to a beefy GPU cluster, world-class mentorship, and free agency to pursue any approach.

I used the machine learning framework XGBoost, cutting-edge deep learning software AlphaFold-Multimer (AF-M), and over 47,000 SLURM jobs to build a multi-classifier model that predicts the effects of missense mutations on PPIs with a 91% AUC.

\"Multi-classMulti-class ROC curve and AUC (By Author)

In this article, I’ll walk through:

If you’re involved in bioinformatics or molecular biology and want to start incorporating machine learning into your research—this article is for you.

Let’s jump into it!

Background

In this section, I’ll cover:

Not so hot take: protein-protein interactions are important.

I might be biased because I studied them for eight months at The Center for Applied Genomics at SickKids—but I’ve got good reason to think so:

So, if we can understand how and why a given mutation impacts PPIs, there’s a lot we can do:

With DeepMind’s release of AlphaFold-Multimer (AF-M), a version of AlphaFold trained with protein complexes in mind, came an opportunity and a question:

Can AF-M capture the effect of missense variants on PPIs?

You might be wondering—why missense variants specifically? As opposed to other kinds of mutations?

Well…

  1. AF-M can’t predict macro-level structural changes from point mutations.
  2. However, in silico changes at the interface of a PPI can have measurable changes (observed through a program like ChimeraX).

It’s a research question that lets us take advantage of current technology.

Now—how do we go about answering this question? I’ll answer that in the next section.

Methodology & Approach

In this section, I’ll cover:

3D structures — like the kind AF-M generates — have exclusive observable properties you can’t get anywhere else.

Most of these are structural features like:

This is exciting data we can only observe from PDB structures. Given the power of a 3D structure, we arrived at the following approach:

  1. Take a wildtype protein complex (just a regular old homo- or hetero-dimer).
  2. Now, take the same complex, except this time with a missense variant in one of the proteins. Effectively, a ‘missense complex.’

If we can detect the difference between the structural features of these complexes and relate it to the specific missense variant, we can find a relationship:

Impact on PPI ~ (structural features).

In other words, impact on PPI as a function of structural features.

\"IllustrationIllustration of wildtype and missense complex comparison

But there’s another opportunity here: what if I enriched that structural data with variant annotations, too? Examples include:

We’d have many more features to parse a relationship from, developing a more comprehensive function:

Impact on PPI ~ f(x₁, x₂, …, xn)

During my data collection process, I was able to add 40+ more non-structural features on top of the structural ones. This made for excellent training data for an XGBoost classifier model that could predict the Impact on PPI of a given variant out of these four classes:

You’re probably wondering—how did I get all this data matching missense variants to PPI effects? And why did I choose XGBoost as my classifier model?

I’ll explain in the following sections.

Data Acquisition & Processing

In this section, I’ll cover:

IntAct Mutations Database

I gathered my training data from the IntAct Mutations database (available under license CC0). It’s a massive annotated record with thousands of missense variants involved in binary protein interactions.Every variant is annotated with its effect on PPI, encoded under the ‘Feature type’ column: increase, decrease, no effect, etc.:

\"DataData encoding in IntAct Mutations database

However, you’ll notice this dataset does NOT include structural feature information or other variant annotations. That’s where I (by which I mean AF-M) came in.

The database contains ~12,000 eligible data points. To save compute and ensure balanced data classes, I randomly subsampled variants until I had ~1,000 for each class. Curious about how random subsampling works?

Random subsampling is when I randomly pull datapoints from a given class until I have enough. This is one approach to ensuring your classes are weighted equally in your training data set so your classifier doesn’t bias towards any single one.Of course, you could also simply assign different weights to each class instead, but random subsampling means I have fewer protein complexes to fold—saving on compute.

\"OverviewOverview of data aggregation pipeline

I then wrote a script that generated FASTA files for the wildtype and missense complexes before feeding them into AF-M to produce PDB structures. Due to time constraints, the final dataset had about 250 structures per class.

\"CreatingCreating a wildtype and missense complex

Wondering how I went from UniProt IDs to PDB structures?This was a relatively simple Bash script that would pull UniProt IDs from the Intact Mutations file and download the appropriate FASTAs via CURL request. We create a wildtype FASTA and then ‘mutate’ a copy to make a missense version. These are inputs for AF-M.

This leaves us with two PDB versions of each protein complex: a wildtype and a missense variant (see image above). Multiply that across ~a thousand binary interactions and 47,000 SLURM jobs, and we have ~20TB worth of data to work with.

Feature Extraction & Engineering

All I had left to do was extract structural and variant feature information:

\"OverviewOverview of data aggregation pipeline, final steps

The AF-M pipeline I used, AlphaPulldown, simulates a pulldown assay by parallelizing the ‘folding’ of several protein complexes. It also includes a feature extraction pipeline that generates several key structural metrics:

And beyond. I also added a few features of my own, with annotations from Variant Effect Predictor by Ensembl, available under license Apache 2.0:

Pathogenicity predictions

Two examples of pathogenicity predictions I used:

Why (and how) I included them:

gnomAD frequencies

gnomAD (the Genome Aggregation Database from the Broad Institute) contains population allele frequencies for several different groups and variants.

Why (and how) I included frequencies:

Relative ratios

I engineered simple ratio features, like the interface area / total surface area, the differences in feature x from the wildtype version, etc.

Why I included relative ratios:

Free energy

Using EvoEF2 (available under the MIT license), I obtained thermodynamic data on protein complexes, comparing 𝚫𝚫G (the difference in Gibbs energy of the folded and unfolded protein states) of the wildtype and mutant variants.

Why free energy:

By the end of it all, I had a tabular dataset that looked a bit like this:

\"Screenshots

One thing about bioinformatics tools, however—they don’t always work. Metrics for a given variant will often be empty or missing. For example, not every variant in IntAct has an associated gnomAD frequency.

Wondering why can missing values make ML challenging?It probably makes intuitive sense as to why missing values are problematic in any machine learning task. Given an incomplete picture of the dataset, it can be hard to trust the output of your model. Luckily, there are methods for dealing with our trust issues.

Missing values are pretty annoying whenever you do machine learning, so I had to pick a model with the right chops. Luckily, XGBoost had what it took—let me explain how.

The Machine Learning Model

In this section, I’ll break down:

XGBoost

XGBoost is a gradient-boosted decision tree model.

Decision trees are graphs that split data at threshold-based nodes. Here’s a simple model of deciding whether or not to play outside today:

\"Decision

Decision tree diagram

Chaining several of these trees together in sequence allows each one to correct on the error of the last one, building a much more robust model. This is what makes it ‘gradient-boosted.’

XGBoost is a lightweight, fast-training model that has several advantages for this particular task:

How does XGBoost handle missing data?The model treats missing data as essentially their own kind of value. At a splitting node, the algorithm determine which direction (left/right) leads to the highest gain (best reduction of the loss function) and encodes that decision into its training. When it encounters a similar missing value in the test set, it’ll make the same decision.

Since the dataset had reasonably sized dimensions and XGBoost trains so quickly, I decided to include a hyperparameter tuning step using k-fold cross-validation:

Hyperparameter tuning & k-fold cross-validation

Any machine learning model will have internal parameters—the weights and biases it ascribes to its internal classifications.

However, ML models also have hyperparameters, which are higher-level structural settings for the overall model. These parameters are set before training and can significantly impact accuracy outcomes.

Here are the ones I focused on:

We determine the optimal hyperparameters by testing several combinations and selecting the one with the best results—in other words, by tuning.

To add even more robustness to the model training, I implemented a k-fold cross-validation step. What is that exactly?

Let’s break it down:

\"By

By MBanuelos22 from Wikimedia Commons, CC BY-SA 4.0

We do this for every hyperparameter combination, ensuring we get the highest accuracy model possible.

Now that we’ve got a model trained and tested, how good is it? What can it teach us about biology? We’ll cover that next:

Results, Model Accuracy & Feature Importances

In this section, I’ll cover:

Confusion matrix & ROC curve

To assess the quality of a multi-classifier, you’ll typically use a confusion matrix:

\"FigureFigure 1. Confusion matrix

The confusion matrix visualizes the test set. Along the bottom, we have what the model predicted for each data point (a protein complex), and on the y-axis, we have the actual value for each data point.The lighter the colour of the diagonal, the more accurate the model.

The confusion matrix is just one way of assessing the model’s accuracy. We can also use an ROC curve:

\"FigureFigure 2. Multi-class ROC curve and AUC

A receiver operating characteristic (ROC) curve plots FPR vs. TPR:

The points along the curve indicate different threshold settings. The threshold refers to the cutoff point where we distinguish between a positive and negative case.

In a multi-class ROC curve, the positive case is a given class, and the negative case is all the others (one-vs-all). The diagonal (dashed line) refers to random chance, where TPR = FPR at all thresholds. One way to think of accuracy is that the further we are from this line, the better the model.

The ideal classifier has a bowed-out shape, where we have a high TPR at low FPR rates, even at high thresholds. We see this for most of our curves.

Microaveraging the curves involved summing each class’s TPs, FNs, and TNs (seen above).

This gives us an overall picture of our model’s performance. To standardize our comparison, we calculate the area under the curve. Closer to 1 = more accurate. As seen in Figure 2, we achieve a micro average AUC of 0.91.

You can also see that the curves for classes 0 and 1 reflect what we saw in the confusion matrix. Overall, we have a reasonably accurate model. But now comes the fun part—what can it tell us about biology?

SHAP values

Using SHAP values can help us determine how much the features in a model contributed to each prediction.

What are SHAP values?Shapley Additive exPlanations measure the marginal contributions of each feature to a class prediction. In short, they measure feature importance. SHAP values are useful because they work for any supervised machine learning task.

They’re determined by:

  1. Taking the average prediction for a given class.
  2. Iteratively adding each feature in (in all possible orders) and evaluating the output.
  3. Calculating the marginal effect of a feature on the model output’s distance from average.

SHAP values are biologically useful because they can tell us which structural features require more study, or which pathogenicity predictions are most accurate.

Here are the top ten feature importances for the entire model:

\"FigureFigure 3. Feature importance

From my analysis, I saw some interesting results.

Realistically, finding one or two features globally predictive of PPI effects is a fruitless exercise. We’re better off looking at feature importance class by class using SHAP values:

Class 0: Decrease in interaction strength

\"Figure\"Figure

On the left, I have a feature importance plot showing the average impact of a feature on the model’s output. This graph is relatively simple to interpret; the bigger the bar, the greater the magnitude of impact.

On the right, we have a beeswarm plot. These kinds of plots help us map two factors:

We can combine these two factors to further our understanding of the feature effects. I’ll use the Resulting_sequence_A feature as an example. (Background: this is a dummy value, so values are either 0 or 1).

In natural language, it seems alanine as a missense residue has a strong effect on decreasing interaction strength. This tracks intuitively—alanine is a small amino acid with only a methyl group for a side chain.

\"Alanine

This limits the kind of interactions it can participate in. So, if it replaces a crucial disulfide bride or prevents a charged interaction, it makes sense that it would increase the likelihood of decreased interaction strength.

Combining wet and computational biology for a better model.While there is a biological case to be made that this finding reflects reality, it’s also true that scientists often conduct alanine-scanning studies where residues are sequentially replaced in a protein sequence to determine which ones are important in folding, interaction, etc.

It’s possible the results from these studies are overrepresented in the IntAct database, biasing the model to think this feature is more important than it is.Interestingly, it was crucial I have background in ‘wet’ biology to understand issues like this. Otherwise, I may not have even realized this was a potential problem.

We see some unexpected results with the shape complementarity (sc) feature. sc measures how well the two proteins fit into each other; we’d expect that when sc is low, it increases the likelihood of decreased interaction strength between two proteins. However, this is not the case: higher sc seems to decrease interaction strength, and vice versa.

In the following sections, I’ll only call out a few features I found interesting:

Class 1: Disruption of interaction

\"Fig.\"Fig.

Class 2: Increase in interaction strength

\"Fig.\"Fig.

Class 3: No effect on interaction

\"Fig. \"Fig.

All in all, some interesting results. Now, let’s apply the model to some novel protein complexes to see if we can learn anything new:

In this section, I’ll:

This is all fun and interesting, but what’s the point of machine learning in molecular biology if we haven’t learned anything new?

So, for the application part of this project, I used the model to analyze a few coding missense variants from the MSSNG database, a repository of whole genome sequence and annotation data designed to study autism spectrum disorder (ASD). (The Scherer Lab specializes in studying the genomics of ASD).

MSSNG is a vast body of data, so I refined my analysis to only a subset that met the following criteria:

Here are two of the standout variants from this subset:

The model predicts Class 1 (a disrupted interaction) for both variants. Let’s take a closer look:

Androgen receptor & tyrosine protein kinase (AR:YES1.p.Q234R)

\"By

Here are the facts:

So here’s one potential story:

Unfortunately, there are a few issues here:

The bottom line is that there are no conclusive results from the model. For your interest, here are the SHAP values that lead to this prediction. Note the comparatively high difference in interface solvation energy from the wildtype.

\"By

Transcriptional coactivator & zinc finger protein (YAP1:GLIS3.p.G448D)

\"By

Once again, here are the facts:

One possible story here: Disrupted interaction between these two proteins may expedite the progression of neuropathologies that result in autism.

However, gleaning a more specific mechanism from this research is a challenge.

Once again, here are its SHAP values:

\"By

So far we’ve trained a reasonably strong model, interpreted its results, and used it to learn more about biology. What are the takeaways? And what’s next?

Final Thoughts and Lessons

Future Directions

I spent about four months at TCAG working on this project, much of which was spent collecting training data. (Not to mention the four months I spent there previously getting accustomed to AF-M).

Given more time, here are steps I’d take to expand the project:

However, I’m quite proud of what I achieved and learned in this timeframe.

Conclusions & Key Takeaways

I can’t imagine many students are given free rein over such an exciting research question, access to world-class mentors, and one of Canada’s largest compute clusters.

But I did.

So, I’m incredibly thankful to everyone involved (Dr. Brett Trost, Dr. Richard Wintle, and Dr. Steve Scherer, to name a few).

Working in the Scherer Lab showed me research doesn’t have to be stale, slow, and tedious. With the right team and vision, it can be fast, dynamic, and cutting-edge.

On a macro-scale, I’ve learned how crucial AI/ML will be to the future of computational biology.

What are your thoughts?

I’m happy to get your feedback, questions, compliments, and curses. Thanks for reading!

", "url": "https://murto.co/xgboost-ppi/", "image": "/img/seo/afm-xgboost-ppi-cover.png", "tags": ["bioinformatics","ml"], "date_published": "2024-05-04T00:00:00+00:00", "date_modified": "2024-05-04T00:00:00+00:00", "author": { "name": "Murto Hilali", "url": "https://murto.co/", "avatar": "https://murto.co/img/author/website-headshot.jpeg" } }, { "id": "https://murto.co/uwaterloo-sickkids-ai/", "title": "Getting Featured by UWaterloo for AI Research at SickKids", "summary": "My briliant friend Kritika and I got to chat about doing AI research at SickKids.", "content_text": "I was lucky enough to join my brilliant friend and colleauge Kritika Grover in getting featured by the University of Waterloo for our work on studying cerebral palsy using AlphaFold at SickKids. You can read the full article here.We were also featured: As part of UWaterloo’s On It campaign In UW Imprint", "content_html": "

\"\"I was lucky enough to join my brilliant friend and colleauge Kritika Grover in getting featured by the University of Waterloo for our work on studying cerebral palsy using AlphaFold at SickKids. You can read the full article here.

We were also featured:

", "url": "https://murto.co/uwaterloo-sickkids-ai/", "image": "/img/seo/uwaterloo-sickkids-ai.png", "date_published": "2024-04-23T00:00:00+00:00", "date_modified": "2024-04-23T00:00:00+00:00", "author": { "name": "Murto Hilali", "url": "https://murto.co/", "avatar": "https://murto.co/img/author/website-headshot.jpeg" } }, { "id": "https://murto.co/nature-publication/", "title": "Co-authorship in Nature Genetics with SickKids", "summary": "Part of my work at SickKids contributed to a research paper in Nature.", "content_text": "OverviewI spent two co-op work terms doing bioinformatics research at The Center for Applied Genomics (TCAG) at the Hospital for Sick Children (SickKids), where I had the opportunity to work on studying how missense variants affected protein-protein interactions.While there, I had the opportunity to contribute to a paper covering the genomic architecture of cerebral palsy. My contributions involved simulating PPI between CP-affected proteins using AlphaFold-Multimer in an attempt to identify any notable differences between wildtype and missense interactions, as well creating a supplemental figure.Here’s a quote from the paper describing the research I worked on: To gain preliminary functional insight into potential molecular mechanisms of missense variants detected in the CP cohort, we used in silico approaches to investigate two potential mechanisms: disruption of protein–protein interactions34 and disruption of phosphorylation-mediated signaling. We investigated three selected de novo P/LP missense variants using AlphaFold-Multimer to assess whether their mechanisms could disrupt protein–protein interactions (p.A334T in TUBA1A, p.D132A in EXOSC3 and p.E237D in GNAO1). These variants were selected based on previous evidence that missense variants in their respective proteins may disrupt protein–protein interactions. TUBA1A and EXOSC3 were not predicted by AlphaFold-Multimer to be involved in the interface with their putative interacting protein(s) (Supplementary Table 8, sheet 1). However, the GNAO1 variant, p.E237D, was predicted by AlphaFold-Multimer to be involved in the interface with its regulator RGS4. It is interesting that a de novo variant at the same position, but causing a different amino acid change (p.E237K), has previously been implicated in CP-like features40. We modeled both missense variants in AlphaFold-Multimer to assess the GNAO1–RGS4 interaction. Despite the different amino acid characteristics of the two missense variants (aspartate versus lysine), they were predicted to have similar effects on the GNAO1–RGS4 interface (Supplementary Table 8, sheet 2 and Supplementary Fig. 3). Furthermore, the p.E237D structure was predicted to be more similar to the p.E237K structure (distance = 0.28) than to the wild-type structure (distance = 0.45).Skills Collaborative research: Before my co-op, I had no idea how much work went into the development of a research paper. Countless meetings, conversations, and SLURM jobs. Independent research and discovery: As one of the only members of the lab working on an interactomics, I laid much of the groundwork for the project myself. Luckily, I had great mentors in Dr. Brett Trost and Dr. Bank Eungchan. AlphaFold-Multimer: AF-M is an incredible tool (especially in conjunction with a tool like PyMol), solving some of the most elusive problems in the world of computational molecular biology. Getting to work with it every day was incredibly illuminating.Key Takeaways There’s a lot left to do: While it may seem like AlphaFold has answered every question we have, it’s opened up even more. Can we design algorithms that predict the effects of structure disrupting/altering point mutations? Can we have them design proteins for us? There’s so much opportunity available. Research takes a long time: I first learned the paper would be getting published in mid-2022. Publication occurred nearly a year later. No complaints, obviously, but it’s a strong reminder of the checks and balances required to put out quality research.", "content_html": "

\"Nature

Overview

I spent two co-op work terms doing bioinformatics research at The Center for Applied Genomics (TCAG) at the Hospital for Sick Children (SickKids), where I had the opportunity to work on studying how missense variants affected protein-protein interactions.

While there, I had the opportunity to contribute to a paper covering the genomic architecture of cerebral palsy. My contributions involved simulating PPI between CP-affected proteins using AlphaFold-Multimer in an attempt to identify any notable differences between wildtype and missense interactions, as well creating a supplemental figure.

Here’s a quote from the paper describing the research I worked on:\"Figure

To gain preliminary functional insight into potential molecular mechanisms of missense variants detected in the CP cohort, we used in silico approaches to investigate two potential mechanisms: disruption of protein–protein interactions34 and disruption of phosphorylation-mediated signaling. We investigated three selected de novo P/LP missense variants using AlphaFold-Multimer to assess whether their mechanisms could disrupt protein–protein interactions (p.A334T in TUBA1A, p.D132A in EXOSC3 and p.E237D in GNAO1). These variants were selected based on previous evidence that missense variants in their respective proteins may disrupt protein–protein interactions. TUBA1A and EXOSC3 were not predicted by AlphaFold-Multimer to be involved in the interface with their putative interacting protein(s) (Supplementary Table 8, sheet 1). However, the GNAO1 variant, p.E237D, was predicted by AlphaFold-Multimer to be involved in the interface with its regulator RGS4. It is interesting that a de novo variant at the same position, but causing a different amino acid change (p.E237K), has previously been implicated in CP-like features40. We modeled both missense variants in AlphaFold-Multimer to assess the GNAO1–RGS4 interaction. Despite the different amino acid characteristics of the two missense variants (aspartate versus lysine), they were predicted to have similar effects on the GNAO1–RGS4 interface (Supplementary Table 8, sheet 2 and Supplementary Fig. 3). Furthermore, the p.E237D structure was predicted to be more similar to the p.E237K structure (distance = 0.28) than to the wild-type structure (distance = 0.45).

Skills

Key Takeaways

", "url": "https://murto.co/nature-publication/", "image": "/img/seo/nature-pub-cover.png", "date_published": "2024-03-29T00:00:00+00:00", "date_modified": "2024-03-29T00:00:00+00:00", "author": { "name": "Murto Hilali", "url": "https://murto.co/", "avatar": "https://murto.co/img/author/website-headshot.jpeg" } }, { "id": "https://murto.co/vici/", "title": "VICI: in silico Controls for PPI Variants", "summary": "An open source command line tool to find P/LP and B/LB controls to test AF-M using PIONEER.", "content_text": "(⌘ or Ctrl + click to follow ↑)As part of my project at SickKids Research Institute testing AlphaFold-Multimer’s capabilities in detecting the effects of missense residues at protein-protein interfaces (PPIs), I needed to find variants of known effect (pathogenic and benign), restricted to those that occur at PPI-encoding sequences.This meant I needed three sets of data: Pathogenic/likely pathogenic variants. If AF-M can detect the impact of variants on PPIs, we’d expect these to have some effect on its predicted structures (positive control). Benign/likely benign variants. Similarly, we’d expect these variants to have no effect on PPIs/structures predicted by AF-M (negative control). Variants known to occur in PPI-encoding sequences. Since I was only interested variants directly involved in PPIs, I needed to restrict the variants in the first two bullets to the ones that also appear in this set.In my research, I was able to easily get the first two sets of data from ClinVar. As well, PIONEER also offers a dataset of human variants occurring at PPI-encoding sequences. I packaged up the code I built to do this task into vici, which takes data from both of these sources (as specified by the user) to output a JSON of positive and negative controls../vici.sh \\-B [path/to/benign/variant/table] \\-P [path/to/pathogenic/variant/table] \\-O [output_folder_name]While I used this to test AlphaFold, it may find other uses in obtaining in silico controls for further study of PPI-perturbing variants. A full description of the pipeline can be found the in the README documentation for the tool.vici currently only accepts ClinVar search data as inputs — which may explain the 0 stars on this repo. However, that means there’s alway opportunity for improvement!", "content_html": "

\"vici

\"GitHub\"GitHub\"License\"\"Stars\"

(⌘ or Ctrl + click to follow ↑)

As part of my project at SickKids Research Institute testing AlphaFold-Multimer’s capabilities in detecting the effects of missense residues at protein-protein interfaces (PPIs), I needed to find variants of known effect (pathogenic and benign), restricted to those that occur at PPI-encoding sequences.

This meant I needed three sets of data:

In my research, I was able to easily get the first two sets of data from ClinVar. As well, PIONEER also offers a dataset of human variants occurring at PPI-encoding sequences. I packaged up the code I built to do this task into vici, which takes data from both of these sources (as specified by the user) to output a JSON of positive and negative controls.

./vici.sh \\-B [path/to/benign/variant/table] \\-P [path/to/pathogenic/variant/table] \\-O [output_folder_name]

While I used this to test AlphaFold, it may find other uses in obtaining in silico controls for further study of PPI-perturbing variants. A full description of the pipeline can be found the in the README documentation for the tool.

vici currently only accepts ClinVar search data as inputs — which may explain the 0 stars on this repo. However, that means there’s alway opportunity for improvement!

", "url": "https://murto.co/vici/", "image": "/img/seo/vici-cover-new.png", "date_published": "2023-09-26T00:00:00+00:00", "date_modified": "2023-09-26T00:00:00+00:00", "author": { "name": "Murto Hilali", "url": "https://murto.co/", "avatar": "https://murto.co/img/author/website-headshot.jpeg" } }, { "id": "https://murto.co/damia/", "title": "DAMIA: Disease-Associated Missense Interactome Analysis", "summary": "Pipeline and command line utility for analyzing missense variant protein-protein interfaces in silico.", "content_text": " Table of Contents Table of Contents Overview Distance Interface Similarity AvgFeatures Interactions Whitney Pipeline Lessons \\& TakeawaysOverviewAs part of my work at SickKids, I needed to compare wildtype protein complexes predicted using AlphaFold-Multimer (AFM) to ones with missense mutations introduced.In conducting this analysis, I created several tools to obtain very specific kinds of information: Tabular representations of interacting residues and chains Average distances between amino acid carbons (alpha & beta) Jaccard similarity between the interacting residues in wildtype and missense complexesMany of these scripts came together to form a pipeline I called DAMIA (Disease-Associated Missense Interactome Analysis) that helps analyze the similarities and differences between wildtype and missense complexes as predicted by AFM.Here’s an overview of each tool, with large reference to the documentation. distance: Distance calculation between alpha and beta carbons interface: Interface determination for chain-residue pairs similarity: Jaccard index similarity calculation avgfeatures: Averages of PI-score output interactions: Interaction search and MFA creation whitney: U-test between WT and missense interfaces.DistanceThis script calculates the distances between the alpha- and beta-carbons of interface residue pairs in a PDB file, using PyMOL.The rationale for the creation of this tool is that we might expect a significant difference in average carbon distance between interacting residues in missense complexes if AFM can truly capture the effects of the mutation.The input is an .ic file containing the list of residue pairs, and the output is a TSV file containing the calculated distances. This provides some information about interface quality and proximity.To run the script from the command line, use the following format:python distance.py <ic_file> <pdb_file> [-o <output_tsv>]Where: <ic_file>: Path to the .ic file containing the residue pairs. <pdb_file>: Path to the PDB file containing the protein structure. <output_tsv>: (Optional) Path to the output TSV file where the distances will be saved. Defaults to output.tsv in the current directory.Example:python distance.py input.ic protein.pdb -o distances.tsvInputThe input .ic file should contain a list of residue pairs, one pair per line. Each line should contain the following columns, separated by whitespace: Residue type of the first residue (e.g., “GLY”). Residue number of the first residue (e.g., “42”). Chain identifier of the first residue (e.g., “A”). Residue type of the second residue (e.g., “LEU”). Residue number of the second residue (e.g., “87”). Chain identifier of the second residue (e.g., “B”).For example:GLY 42 A LEU 87 BOutputThe output TSV file will contain the following columns: Index: The index of the residue pairn from the .ic file. Chain1: Chain identifier of the first residue. Res1: Residue type of the first residue. Res1_num: Residue number of the first residue. Chain2: Chain identifier of the second residue. Res2: Residue type of the second residue. Res2_num: Residue number of the second residue. Alpha_distance: Distance between the alpha-carbons of the two residues. Beta_distance: Distance between the beta-carbons of the two residues, or “N/A” if one of the residues is glycine.Example output:index chain1 res1 res1_num chain2 res2 res2_num alpha_distance beta_distance1 A GLY 42 B LEU 87 12.34 N/ACode OverviewThe script is divided into the following functions: calculate_distance: Calculates the distance between the alpha- and beta-carbons of two residues in a PDB file. main: Reads the input .ic file, calculates distances for all residue pairs, and writes the results to the output TSV file.The script uses the argparse library to parse command-line arguments, and the pymol library to load PDB files and calculate distances.InterfaceThis Python script analyzes occurrences of a chain-residue pair across multiple TSV files and calculates a consensus score based on their presence. The script also calculates the average distance (either alpha or beta) between the chain-residue pairs in the provided TSV files. interface.py aims to show often a given residue (at a specified chain) appears in interfaces and, when it does, how far it is, on average, from the other residues it interacts with.The rationale for this tool’s existence is to observe interface residues as they appear across AFM predictions (the algorithm can produce up to 5 per run, per sequence).Usagepython interface.py <file_paths> <chain> <residue> [-d <distance_type>]Arguments file_paths: Paths to the TSV files (space-separated). chain: Chain (a letter A-Z). residue: Residue (an integer). -d, --distance_type: (Optional) Distance type, either “alpha” or “beta”. Default is “beta”.ExampleSuppose we have two TSV files file1.tsv and file2.tsv. We want to analyze the occurrences of the chain-residue pair A-42 with alpha distance. The command would look like:python interface.py file1.tsv file2.tsv A 42 -d alphaOutputThe script prints the consensus score and total occurrences of the chain-residue pair to the console. Additionally, it generates two output files: occurrences.tsv: A TSV file containing the occurrences and average distances for each input TSV file. avg_distance.txt: A plain text file containing the overall average distance.Functionsread_tsv(file_path: str) -> pd.DataFrameReads a TSV file and returns a pandas DataFrame.find_chain_residue(df: pd.DataFrame, chain: str, residue: int, distance_type: str) -> (bool, int, float)Finds the occurrences of a chain-residue pair in the given DataFrame and calculates the average distance based on the specified distance type. Returns a tuple containing a boolean indicating presence, the number of occurrences, and the average distance.Example Input TSV FileThe input TSV files should have the following format (this comes from PRODIGY):index chain1 res1 res1_num chain2 res2 res2_num alpha_distance beta_distance1 A ALA 15 B GLY 32 12.34 10.562 A VAL 42 B MET 29 7.92 6.343 B ALA 15 A GLY 32 11.87 9.42Example Output TSV FileThe output TSV file occurrences.tsv will have the following format:TSV Occurrences Average Distancefile1.tsv 2 6.45file2.tsv 1 7.92Example Output TXT FileThe output plain text file avg_distance.txt will have the following content:Average Distance: 6.94SimilarityThis script calculates the Jaccard Index similarity between multiple TSV files, each containing interaction data from a complex. Users can provide options to keep only the residue numbers, remove duplicate residues, or both. The script saves the pairwise similarity scores in a TSV file and writes a summary of the results to a text file.similarity.py tells us how similar the interfaces of different complexes are to each other.Command-Line Arguments file_paths (str): Paths to the TSV files to be compared. --residues_only (bool, optional): If provided, keep only the residue number columns in the input DataFrames. --remove_duplicate_residues (bool, optional): If provided, remove duplicate residues from the input DataFrames.Example UsageHere is an example of how to run the script from the command line:python similarity.py file1.tsv file2.tsv file3.tsv --residues_only --remove_duplicate_residuesIn this example, the script will: Read the contents of file1.tsv, file2.tsv, and file3.tsv. Preprocess each DataFrame by keeping only the residue number columns (res1_num and res2_num) and removing duplicate residues if the --residues_only and --remove_duplicate_residues flags are provided. Calculate the pairwise Jaccard Index similarity between each pair of tables. Save the pairwise similarity scores in a TSV file named pairwise_similarity.tsv. Calculate the average, median, minimum, and maximum similarity scores and save the summary in a text file named summary.txt.The output files pairwise_similarity.tsv and summary.txt will be generated in the same directory as the script.Functions read_tsv(file_path: str) -> pd.DataFrame: Reads a TSV file and returns a pandas DataFrame. file_path (str): The file path of the TSV file. preprocess_table(df: pd.DataFrame, residues_only: bool, remove_duplicate_residues: bool) -> pd.DataFrame: Preprocesses a DataFrame by keeping only the residue number columns or removing unnecessary columns and duplicate rows. df (pd.DataFrame): The input DataFrame. residues_only (bool): If True, keep only the residue number columns. remove_duplicate_residues (bool): If True, remove duplicate residues. jaccard_index(df1: pd.DataFrame, df2: pd.DataFrame) -> float: Calculates the Jaccard Index similarity between two DataFrames. df1 (pd.DataFrame): The first DataFrame. df2 (pd.DataFrame): The second DataFrame. multi_table_jaccard(file_paths: List[str], residues_only: bool, remove_duplicate_residues: bool) -> Tuple[pd.DataFrame, pd.Series]: Calculates pairwise Jaccard Index similarity between multiple TSV files and returns a DataFrame with the results and a summary Series. file_paths (List[str]): A list of file paths to the TSV files. residues_only (bool): If True, keep only the residue number columns in the input DataFrames. remove_duplicate_residues (bool): If True, remove duplicate residues from the input DataFrames. Output Files pairwise_similarity.tsv: A TSV file containing the pairwise similarity scores. summary.txt: A text file containing the average, median, minimum, and maximum similarity scores.Example Input:Let’s consider three example TSV files, file1.tsv, file2.tsv, and file3.tsv.file1.tsv:index chain1 res1 res1_num chain2 res2 res2_num alpha_distance beta_distance1 A ALA 15 B GLY 32 12.34 10.562 A VAL 42 B MET 29 7.92 6.343 B ALA 15 A GLY 32 11.87 9.42file2.tsv:index chain1 res1 res1_num chain2 res2 res2_num alpha_distance beta_distance1 A ALA 15 B GLY 32 13.45 11.322 B VAL 42 A MET 29 8.01 6.474 A ARG 8 B LEU 21 10.29 8.61file3.tsv:index chain1 res1 res1_num chain2 res2 res2_num alpha_distance beta_distance1 A ALA 15 B GLY 32 12.78 10.892 A VAL 42 B MET 29 7.95 6.383 B ALA 15 A GLY 32 11.89 9.445 A ARG 8 B LEU 21 10.31 8.63Command:python multi_table_jaccard.py file1.tsv file2.tsv file3.tsv --residues_only --remove_duplicate_residuesExample Output:pairwise_similarity.tsv:Pairwise Matchup Similarity ScoreTable 1 - Table 2 0.6Table 1 - Table 3 0.75Table 2 - Table 3 0.6summary.txt:Average Score: 0.65Median Score: 0.6Minimum Score: 0.6Maximum Score: 0.75In this example, the pairwise similarity scores between the tables are 0.6 and 0.75, indicating that the residue pairs in the tables are not identical after preprocessing.AvgFeaturesThis Python script reads a CSV file containing PI-Score features and calculates the averages, standard deviations, and standard errors for each numeric feature. The results are saved in a TSV file.Usagepython avgfeatures.py <csv_path> [-o output_file]Arguments csv_path: Path to the input CSV file. output_file: (Optional) Output file name. Default is “averages.tsv”.ExampleSuppose we have a CSV file input.csv. We want to calculate the averages, standard deviations, and standard errors for the numeric features and save the results in a file called “results.tsv”. The command would look like:python avgfeatures.py input.csv -o results.tsvOutputThe script generates an output TSV file containing the averages, standard deviations, and standard errors for each numeric feature in the input CSV file.Functionsprocess_csv_file(filename: str, output_filename: str)Reads a CSV file, calculates the averages, standard deviations, and standard errors for each numeric feature, and saves the results in a TSV file.Example Input CSV FileThe input CSV files should have the following format:pdb,interface,Num_intf_residues,Polar,Hydrophobhic,Charged,conserved_interface, contact_pairs, sc, hb, sb, int_solv_en, int_area, pvalueranked_0,B_A,54,0.278,0.259,0.37,NA,55,0.629,21,6,-8.41,1770.08,0.63Example Output TSV FileThe output TSV file will have the following format:Feature\t Average Std. Deviation\tStd. ErrorNum_intf_residues 54.000000\t0.000000\t 0.000000Polar\t 0.278000\t0.000000\t 0.000000Hydrophobhic 0.259000\t0.000000\t 0.000000Charged 0.370000\t0.000000\t 0.000000Note that the example output file contains only a single row of data, as there is only one input row in the example input file. In practice, the input CSV file will have multiple rows, and the output file will show the calculated values based on all input rows.InteractionsThis script fetches protein interaction data from MINT or BioGRID and generates multi-FASTA files for each pair of interaction partners. Its purpose is to create MFA inputs for AF2-multimer to make complex predictions.Usagepython interactions.py <uniprot_ids> --source <mint|biogrid> [--output_folder <output_folder>] [--access_key <access_key>]Arguments uniprot_ids: List of UniProt IDs to fetch data for. --source: Choose the data source (mint or biogrid). Only MINT is currently working. --output_folder (optional): Output folder for interaction data (default: interactions). --access_key (optional): Access key for BioGRID API (required if source is biogrid).Output The script creates a folder with interaction data files for each UniProt ID in the specified output_folder (default: interactions). A JSON file interaction_partners.json is created in the output_folder, containing a dictionary with UniProt IDs as keys and a list of interaction partner UniProt IDs as values. A folder named complexes is created with multi-FASTA files for each pair of interaction partners. The file names are in the format key:value[0].fa, key:value[1].fa, etc.Limitations The .fa files in the complexes folder are automatically generated and do not account for other proteins that may be required to properly model a complex. They also assume all complexes are heterodimeric, which may not be the case. Some of the complexes may have exact experimental structures in the PDB, which would be preferred for most analyses.Functionsget_mint_data(output_folder, uniprot_ids)Fetches interaction data from MINT and saves it as TSV files in the specified output folder.extract_interaction_partners_mint(uniprot_ids, output_folder)Extracts interaction partner UniProt IDs from MINT data and returns a dictionary with UniProt IDs as keys and a list of interaction partner UniProt IDs as values.get_biogrid_data(output_folder, uniprot_ids, access_key)Fetches interaction data from BioGRID and saves it as TSV files in the specified output folder. Also produces interaction_partners.json.extract_interaction_partners_biogrid(uniprot_ids, output_folder)Converts BioGRID gene names to UniProt IDs and returns a dictionary with UniProt IDs as keys and a list of interaction partner UniProt IDs as values.create_mfa(interaction_partners: dict)Generates multi-FASTA files for each pair of interaction partners and saves them in a folder named complexes.main()The main function of the script. It parses command-line arguments, calls the appropriate data fetching and processing functions depending on the selected data source, and generates the JSON file with the interaction partners dictionary.Example UsageTo fetch MINT interaction data for UniProt IDs Q15113, P20908, and Q9BQB4, and save the output in the “interactions” folder, run the following command:python interactions.py Q15113 P20908 Q9BQB4 --source mintThis will create the following files and folders: interactions/Q15113.tsv: MINT interaction data for Q15113 interactions/P20908.tsv: MINT interaction data for P20908 interactions/Q9BQB4.tsv: MINT interaction data for Q9BQB4 interactions/interaction_partners.json: A JSON file containing the dictionary of interaction partners complexes/: A folder containing multi-FASTA files for each pair of interaction partnersSimilarly for BioGRID, run the following command:python interactions.py Q15113 P20908 Q9BQB4 --source biogrid --access_key \"accesskey\"You can obtain an access key from the BioGRID website.Notes The script currently generates multi-FASTA files for each pair of interaction partners without accounting for other proteins that may be required to properly model a complex. The script also assumes all complexes are heterodimeric, which may not be the case. Output folder issue fixed (Thank you Kritika Grover) Some of the complexes may have exact experimental structures in the PDB, which would be preferred for most analyses.TODO The extract_interaction_partners_{mint, biogrid} functions use much of the same code, and can probably be combined. Include a means of checking PDB for existing experimental model.WhitneyThis Python script performs a Mann-Whitney U-test on specified columns from multiple TSV files to determine if the difference between interaction data of two sets of models is significant.Usagepython whitney.py -a <sample_a_files> -b <sample_b_files> -c <column_name>Arguments -a, --sample_a: List of TSV files for sample A (space-separated). -b, --sample_b: List of TSV files for sample B (space-separated). -c, --column_name: Column name to extract data from.ExampleSuppose we have four TSV files, sample_a1.tsv, sample_a2.tsv, sample_b1.tsv, and sample_b2.tsv. We want to perform a Mann-Whitney U-test on the “interaction_strength” column. The command would look like:python whitney.py -a sample_a1.tsv sample_a2.tsv -b sample_b1.tsv sample_b2.tsv -c interaction_strengthOutputThe script prints the Mann-Whitney U-test results, including the U statistic and P-value, to the console. Additionally, it generates an output text file called “whitney.txt” containing the same information.Functionsread_and_extract_column(filepaths: List[str], column_name: str) -> List[float]Reads the specified column from each TSV file in the list of file paths, ignoring rows with “N/A” in the specified column, and returns the combined data as a list of floats.main(sample_a_files: List[str], sample_b_files: List[str], column_name: str)Performs a Mann-Whitney U-test on the specified columns from the given TSV files for samples A and B, prints the results, and writes the results to an output file.Input TSV File FormatThe input TSV files should have the following format:pdb_id interaction_strengthmodel1 0.65model2 0.72model3 0.61Example Output TXT FileThe output text file whitney.txt will have the following content:Mann-Whitney U-test results:U statistic: 15.0P-value: 0.0389249472108PipelineI want to be clear: the ultimate vision for DAMIA is yet to be realized. It’s not an autonomous pipeline just yet. However, all of its scripts can be used independently in a pipeline of sorts. However, the ultimate vision is for DAMIA to function as a simple to use pipeline where all you need to provide is a UniProt ID (or several, in a high-throughput analysis): Start with a UniProt ID input into interactions.py to get the MFAs we need (wildtype and missense). Predict their structures using AFM. Run the PDBs through PRODIGY to get an .ic file. Run the .ic file through distance.py to obtain an enriched TSV file version, with additional carbon distances. Run the TSV files through the whitney.py to see if there’s a statistically significant difference between the wildtype and missense complexes. Run the TSV files through interface.py to see the average distance between the interface residues and how often certain residues occur. If the missense residue being analyzed is in the interface: Use partners.py to find that residues interaction partners. Use avgdistances.py to find average distances between interface residues. Run the complex through PI-score and avgfeatures.py to get the average of all features from that pipelines. Lessons & Takeaways Save your work as you go: This started as a mostly exploratory project. But since I saved all my scripts, I ultimately realized I could package up my workflow into something useful. Plan your pipeline and analysis ahead of time: If these scripts seem a little tricky to follow, it’s because they are. Much of DAMIA was built by going and back and forth in the research process (something I like to call Agile Research). While this is good for quickly iterating on findings, it doesn’t necessary translate well over to novel pipelines like this one.The work I did on this project, culminating in DAMIA, was what led to my contributions in the Nature Genetics paper I write about here, so it means a lot to me!", "content_html": "

\"GitHub \"GitHub \"License\" \"Stars\"

Table of Contents

Overview

As part of my work at SickKids, I needed to compare wildtype protein complexes predicted using AlphaFold-Multimer (AFM) to ones with missense mutations introduced.

In conducting this analysis, I created several tools to obtain very specific kinds of information:

Many of these scripts came together to form a pipeline I called DAMIA (Disease-Associated Missense Interactome Analysis) that helps analyze the similarities and differences between wildtype and missense complexes as predicted by AFM.

Here’s an overview of each tool, with large reference to the documentation.

Distance

This script calculates the distances between the alpha- and beta-carbons of interface residue pairs in a PDB file, using PyMOL.

The rationale for the creation of this tool is that we might expect a significant difference in average carbon distance between interacting residues in missense complexes if AFM can truly capture the effects of the mutation.

The input is an .ic file containing the list of residue pairs, and the output is a TSV file containing the calculated distances. This provides some information about interface quality and proximity.

To run the script from the command line, use the following format:

python distance.py <ic_file> <pdb_file> [-o <output_tsv>]

Where:

Example:

python distance.py input.ic protein.pdb -o distances.tsv

Input

The input .ic file should contain a list of residue pairs, one pair per line. Each line should contain the following columns, separated by whitespace:

  1. Residue type of the first residue (e.g., “GLY”).
  2. Residue number of the first residue (e.g., “42”).
  3. Chain identifier of the first residue (e.g., “A”).
  4. Residue type of the second residue (e.g., “LEU”).
  5. Residue number of the second residue (e.g., “87”).
  6. Chain identifier of the second residue (e.g., “B”).

For example:

GLY 42 A LEU 87 B

Output

The output TSV file will contain the following columns:

  1. Index: The index of the residue pairn from the .ic file.
  2. Chain1: Chain identifier of the first residue.
  3. Res1: Residue type of the first residue.
  4. Res1_num: Residue number of the first residue.
  5. Chain2: Chain identifier of the second residue.
  6. Res2: Residue type of the second residue.
  7. Res2_num: Residue number of the second residue.
  8. Alpha_distance: Distance between the alpha-carbons of the two residues.
  9. Beta_distance: Distance between the beta-carbons of the two residues, or “N/A” if one of the residues is glycine.

Example output:

index   chain1  res1    res1_num chain2  res2    res2_num alpha_distance    beta_distance1       A       GLY     42       B       LEU     87       12.34             N/A

Code Overview

The script is divided into the following functions:

The script uses the argparse library to parse command-line arguments, and the pymol library to load PDB files and calculate distances.

Interface

This Python script analyzes occurrences of a chain-residue pair across multiple TSV files and calculates a consensus score based on their presence. The script also calculates the average distance (either alpha or beta) between the chain-residue pairs in the provided TSV files. interface.py aims to show often a given residue (at a specified chain) appears in interfaces and, when it does, how far it is, on average, from the other residues it interacts with.

The rationale for this tool’s existence is to observe interface residues as they appear across AFM predictions (the algorithm can produce up to 5 per run, per sequence).

Usage

python interface.py <file_paths> <chain> <residue> [-d <distance_type>]

Arguments

Example

Suppose we have two TSV files file1.tsv and file2.tsv. We want to analyze the occurrences of the chain-residue pair A-42 with alpha distance. The command would look like:

python interface.py file1.tsv file2.tsv A 42 -d alpha

Output

The script prints the consensus score and total occurrences of the chain-residue pair to the console. Additionally, it generates two output files:

  1. occurrences.tsv: A TSV file containing the occurrences and average distances for each input TSV file.
  2. avg_distance.txt: A plain text file containing the overall average distance.

Functions

read_tsv(file_path: str) -> pd.DataFrame

Reads a TSV file and returns a pandas DataFrame.

find_chain_residue(df: pd.DataFrame, chain: str, residue: int, distance_type: str) -> (bool, int, float)

Finds the occurrences of a chain-residue pair in the given DataFrame and calculates the average distance based on the specified distance type. Returns a tuple containing a boolean indicating presence, the number of occurrences, and the average distance.

Example Input TSV File

The input TSV files should have the following format (this comes from PRODIGY):

index   chain1  res1    res1_num chain2  res2    res2_num alpha_distance  beta_distance1       A       ALA     15       B       GLY     32       12.34           10.562       A       VAL     42       B       MET     29       7.92            6.343       B       ALA     15       A       GLY     32       11.87           9.42

Example Output TSV File

The output TSV file occurrences.tsv will have the following format:

TSV     Occurrences  Average Distancefile1.tsv       2           6.45file2.tsv       1           7.92

Example Output TXT File

The output plain text file avg_distance.txt will have the following content:

Average Distance: 6.94

Similarity

This script calculates the Jaccard Index similarity between multiple TSV files, each containing interaction data from a complex. Users can provide options to keep only the residue numbers, remove duplicate residues, or both. The script saves the pairwise similarity scores in a TSV file and writes a summary of the results to a text file.

similarity.py tells us how similar the interfaces of different complexes are to each other.

Command-Line Arguments

Example Usage

Here is an example of how to run the script from the command line:

python similarity.py file1.tsv file2.tsv file3.tsv --residues_only --remove_duplicate_residues

In this example, the script will:

  1. Read the contents of file1.tsv, file2.tsv, and file3.tsv.
  2. Preprocess each DataFrame by keeping only the residue number columns (res1_num and res2_num) and removing duplicate residues if the --residues_only and --remove_duplicate_residues flags are provided.
  3. Calculate the pairwise Jaccard Index similarity between each pair of tables.
  4. Save the pairwise similarity scores in a TSV file named pairwise_similarity.tsv.
  5. Calculate the average, median, minimum, and maximum similarity scores and save the summary in a text file named summary.txt.

The output files pairwise_similarity.tsv and summary.txt will be generated in the same directory as the script.

Functions

  1. read_tsv(file_path: str) -> pd.DataFrame:

    Reads a TSV file and returns a pandas DataFrame.

  2. preprocess_table(df: pd.DataFrame, residues_only: bool, remove_duplicate_residues: bool) -> pd.DataFrame:

    Preprocesses a DataFrame by keeping only the residue number columns or removing unnecessary columns and duplicate rows.

  3. jaccard_index(df1: pd.DataFrame, df2: pd.DataFrame) -> float:

    Calculates the Jaccard Index similarity between two DataFrames.

  4. multi_table_jaccard(file_paths: List[str], residues_only: bool, remove_duplicate_residues: bool) -> Tuple[pd.DataFrame, pd.Series]:

    Calculates pairwise Jaccard Index similarity between multiple TSV files and returns a DataFrame with the results and a summary Series.

Output Files

  1. pairwise_similarity.tsv: A TSV file containing the pairwise similarity scores.
  2. summary.txt: A text file containing the average, median, minimum, and maximum similarity scores.

Example Input:

Let’s consider three example TSV files, file1.tsv, file2.tsv, and file3.tsv.

file1.tsv:

index   chain1  res1    res1_num chain2  res2    res2_num alpha_distance  beta_distance1       A       ALA     15       B       GLY     32       12.34           10.562       A       VAL     42       B       MET     29       7.92            6.343       B       ALA     15       A       GLY     32       11.87           9.42

file2.tsv:

index   chain1  res1    res1_num chain2  res2    res2_num alpha_distance  beta_distance1       A       ALA     15       B       GLY     32       13.45           11.322       B       VAL     42       A       MET     29       8.01            6.474       A       ARG     8        B       LEU     21       10.29           8.61

file3.tsv:

index   chain1  res1    res1_num chain2  res2    res2_num alpha_distance  beta_distance1       A       ALA     15       B       GLY     32       12.78           10.892       A       VAL     42       B       MET     29       7.95            6.383       B       ALA     15       A       GLY     32       11.89           9.445       A       ARG     8        B       LEU     21       10.31           8.63

Command:

python multi_table_jaccard.py file1.tsv file2.tsv file3.tsv --residues_only --remove_duplicate_residues

Example Output:

pairwise_similarity.tsv:

Pairwise Matchup      Similarity ScoreTable 1 - Table 2     0.6Table 1 - Table 3     0.75Table 2 - Table 3     0.6

summary.txt:

Average Score: 0.65Median Score: 0.6Minimum Score: 0.6Maximum Score: 0.75

In this example, the pairwise similarity scores between the tables are 0.6 and 0.75, indicating that the residue pairs in the tables are not identical after preprocessing.

AvgFeatures

This Python script reads a CSV file containing PI-Score features and calculates the averages, standard deviations, and standard errors for each numeric feature. The results are saved in a TSV file.

Usage

python avgfeatures.py <csv_path> [-o output_file]

Arguments

Example

Suppose we have a CSV file input.csv. We want to calculate the averages, standard deviations, and standard errors for the numeric features and save the results in a file called “results.tsv”. The command would look like:

python avgfeatures.py input.csv -o results.tsv

Output

The script generates an output TSV file containing the averages, standard deviations, and standard errors for each numeric feature in the input CSV file.

Functions

process_csv_file(filename: str, output_filename: str)

Reads a CSV file, calculates the averages, standard deviations, and standard errors for each numeric feature, and saves the results in a TSV file.

Example Input CSV File

The input CSV files should have the following format:

pdb,interface,Num_intf_residues,Polar,Hydrophobhic,Charged,conserved_interface, contact_pairs, sc, hb, sb, int_solv_en, int_area, pvalueranked_0,B_A,54,0.278,0.259,0.37,NA,55,0.629,21,6,-8.41,1770.08,0.63

Example Output TSV File

The output TSV file will have the following format:

Feature\t        Average    Std. Deviation\tStd. ErrorNum_intf_residues 54.000000\t0.000000\t    0.000000Polar\t          0.278000\t0.000000\t    0.000000Hydrophobhic      0.259000\t0.000000\t    0.000000Charged           0.370000\t0.000000\t    0.000000

Note that the example output file contains only a single row of data, as there is only one input row in the example input file. In practice, the input CSV file will have multiple rows, and the output file will show the calculated values based on all input rows.

Interactions

This script fetches protein interaction data from MINT or BioGRID and generates multi-FASTA files for each pair of interaction partners. Its purpose is to create MFA inputs for AF2-multimer to make complex predictions.

Usage

python interactions.py <uniprot_ids> --source <mint|biogrid> [--output_folder <output_folder>] [--access_key <access_key>]

Arguments

Output

Limitations

Functions

get_mint_data(output_folder, uniprot_ids)

Fetches interaction data from MINT and saves it as TSV files in the specified output folder.

extract_interaction_partners_mint(uniprot_ids, output_folder)

Extracts interaction partner UniProt IDs from MINT data and returns a dictionary with UniProt IDs as keys and a list of interaction partner UniProt IDs as values.

get_biogrid_data(output_folder, uniprot_ids, access_key)

Fetches interaction data from BioGRID and saves it as TSV files in the specified output folder. Also produces interaction_partners.json.

extract_interaction_partners_biogrid(uniprot_ids, output_folder)

Converts BioGRID gene names to UniProt IDs and returns a dictionary with UniProt IDs as keys and a list of interaction partner UniProt IDs as values.

create_mfa(interaction_partners: dict)

Generates multi-FASTA files for each pair of interaction partners and saves them in a folder named complexes.

main()

The main function of the script. It parses command-line arguments, calls the appropriate data fetching and processing functions depending on the selected data source, and generates the JSON file with the interaction partners dictionary.

Example Usage

To fetch MINT interaction data for UniProt IDs Q15113, P20908, and Q9BQB4, and save the output in the “interactions” folder, run the following command:

python interactions.py Q15113 P20908 Q9BQB4 --source mint

This will create the following files and folders:

Similarly for BioGRID, run the following command:

python interactions.py Q15113 P20908 Q9BQB4 --source biogrid --access_key \"accesskey\"

You can obtain an access key from the BioGRID website.

Notes

TODO

Whitney

This Python script performs a Mann-Whitney U-test on specified columns from multiple TSV files to determine if the difference between interaction data of two sets of models is significant.

Usage

python whitney.py -a <sample_a_files> -b <sample_b_files> -c <column_name>

Arguments

Example

Suppose we have four TSV files, sample_a1.tsv, sample_a2.tsv, sample_b1.tsv, and sample_b2.tsv. We want to perform a Mann-Whitney U-test on the “interaction_strength” column. The command would look like:

python whitney.py -a sample_a1.tsv sample_a2.tsv -b sample_b1.tsv sample_b2.tsv -c interaction_strength

Output

The script prints the Mann-Whitney U-test results, including the U statistic and P-value, to the console. Additionally, it generates an output text file called “whitney.txt” containing the same information.

Functions

read_and_extract_column(filepaths: List[str], column_name: str) -> List[float]

Reads the specified column from each TSV file in the list of file paths, ignoring rows with “N/A” in the specified column, and returns the combined data as a list of floats.

main(sample_a_files: List[str], sample_b_files: List[str], column_name: str)

Performs a Mann-Whitney U-test on the specified columns from the given TSV files for samples A and B, prints the results, and writes the results to an output file.

Input TSV File Format

The input TSV files should have the following format:

pdb_id  interaction_strengthmodel1  0.65model2  0.72model3  0.61

Example Output TXT File

The output text file whitney.txt will have the following content:

Mann-Whitney U-test results:U statistic: 15.0P-value: 0.0389249472108

Pipeline

\"\"

I want to be clear: the ultimate vision for DAMIA is yet to be realized. It’s not an autonomous pipeline just yet. However, all of its scripts can be used independently in a pipeline of sorts. However, the ultimate vision is for DAMIA to function as a simple to use pipeline where all you need to provide is a UniProt ID (or several, in a high-throughput analysis):

  1. Start with a UniProt ID input into interactions.py to get the MFAs we need (wildtype and missense).
  2. Predict their structures using AFM.
  3. Run the PDBs through PRODIGY to get an .ic file.
  4. Run the .ic file through distance.py to obtain an enriched TSV file version, with additional carbon distances.
  5. Run the TSV files through the whitney.py to see if there’s a statistically significant difference between the wildtype and missense complexes.
  6. Run the TSV files through interface.py to see the average distance between the interface residues and how often certain residues occur.
  7. If the missense residue being analyzed is in the interface:

Lessons & Takeaways

The work I did on this project, culminating in DAMIA, was what led to my contributions in the Nature Genetics paper I write about here, so it means a lot to me!

", "url": "https://murto.co/damia/", "image": "/img/seo/damia-cover.png", "tags": ["bioinformatics","tools"], "date_published": "2023-04-30T00:00:00+00:00", "date_modified": "2023-04-30T00:00:00+00:00", "author": { "name": "Murto Hilali", "url": "https://murto.co/", "avatar": "https://murto.co/img/author/website-headshot.jpeg" } }, { "id": "https://murto.co/idiomics/", "title": "Idiomics: Handdrawn Idiom Illustrations", "summary": "A collection of free idiom illustrations for blogs and newsletters.", "content_text": "Idiomics is a collection of illustrations I made during the pandemic to help add a bit of life to the articles I was writing on Medium.Throwing the baby out with the bathwater.This was also my first ever launch on Product Hunt — can’t really say you’re in tech without one of these, right? SkillsThroughout the whole process, I managed to practice a few important skills: eCommerce copywriting and landing page design Promotional video creation Product Hunt (yes, this is a skill)Not to mention the actual drawing itself. I’ve made all the illustrations free to download and use.Key TakeawaysI think there’s a lot I learned from building Idiomics. Here are my key takeaways: Sharing is caring: If you do something valuable for yourself, it’s probably valuable to other people too. Just ship it: I had this project on the backburner for about 2 and a half years before I finall pressed publish. Ship! Selling illustrations is hard: This project was never really meant to generate revenue (and trust me — it didn’t). But it’s because it didn’t solve a pressing enough problem.", "content_html": "

\"\"

\"Idiomics

Idiomics is a collection of illustrations I made during the pandemic to help add a bit of life to the articles I was writing on Medium.

\"ThrowingThrowing the baby out with the bathwater.

This was also my first ever launch on Product Hunt — can’t really say you’re in tech without one of these, right?

Skills

Throughout the whole process, I managed to practice a few important skills:

Not to mention the actual drawing itself. I’ve made all the illustrations free to download and use.

Key Takeaways

I think there’s a lot I learned from building Idiomics. Here are my key takeaways:

", "url": "https://murto.co/idiomics/", "image": "/img/seo/idiomics-cover.jpg", "tags": ["fun","writing"], "date_published": "2022-03-24T00:00:00+00:00", "date_modified": "2022-03-24T00:00:00+00:00", "author": { "name": "Murto Hilali", "url": "https://murto.co/", "avatar": "https://murto.co/img/author/website-headshot.jpeg" } } ] }