format

zombi2 tools format DIR --format FORMAT reads a genomes run and writes files derived from its gene trees, beside the run under genomes/. --format is required and takes several at once: --format homology recphylo. Family and ordered runs rebuild their gene trees from the event log; a nucleotide run recovers them from the genome, one file per declared gene (the intergenic spacer is not a gene, so it gets none). A nucleotide run that declared no genes at all is one uninterrupted intergene, and format refuses it: there is nothing to relate. --from PATH reads a run elsewhere, and DIR is then only the output; --flat writes straight into it rather than under genomes/. A declared gene's file is numbered by its family, like everything else, and result.gene_names maps the numbers back to the annotation's names.

File	What it holds
`homology_fam<f>.tsv`	one n×n table per family (n the extant leaves), in `genomes/homology/`. Only with `--format homology`
`markers.tsv`	one row per family that left a surviving copy, a single file at `genomes/markers.tsv`. Only with `--format markers`
`recphylo_fam<f>.xml`	one file per family, in `genomes/recphylo/`: that family's complete gene tree written inside the complete species tree. Only with `--format recphylo`
`recphylo_fam<f>_{true,recoverable}.xml` · `family_origins.tsv`	the same history projected onto what a dataset holds. Only with `--format recphylo --recphylo extant`

homology_fam<f>.tsv. Row and column headers are the leaves n<species>_g<copy>; each off-diagonal cell says how that pair diverged and whether transfer is in its history since, as S, D or T, each optionally with an x (see below). The diagonal is -. Symmetric. A family with no surviving copy writes no table.

markers.tsv. Is the family single-copy, is it universal, and does its true tree match the species tree.

recphylo_fam<f>.xml. Written for every family, extinct ones included, in the recPhyloXML format.

The extant-only pair. The extant gene tree inside the extant species tree, written twice, and a table saying how each family entered.

These files are exact, not inferred: ZOMBI2 recorded each gene tree's embedding in the species tree rather than reconstructing it, which is what makes them an answer key to score a method against.

The homology matrix¶

A cell carries two independent facts about a pair of genes.

How they diverged: the event at their most-recent common ancestor. S a speciation, one gene whose species split in two; D a duplication, one gene became two inside one genome; T a transfer, one gene became two in two different lineages.

Whether transfer is in their history since: an x suffix, when a transfer sits on the path from that common ancestor down to either gene. A pair that diverged at a transfer is already T, so Tx would mean a second one further down.

The smallest interesting case: species ((a, b), c), where c donates a copy of its gene into a, so a ends up holding two.

          n3_g1    n4_g2    n2_g3    n3_g4      n3 = a,  n4 = b,  n2 = c
  n3_g1       -        S        S       Sx      g4 = the copy c sent to a
  n4_g2       S        -        S       Sx      g3 = the copy c kept
  n2_g3       S        S        -        T
  n3_g4      Sx       Sx        T        -

Every pair that reaches the arrival g4 across the transfer carries the x, and no other pair has one. The pair g3–g4, which diverged at the transfer, is the T itself. n3_g1 against n3_g4, two genes in the same genome, reads Sx: they diverged when a and c split, and one came back by transfer.

The table is read off each family's complete gene tree rather than the pruned one. The pairs and the letters would come out the same on either, but the x is a fact about the path: a pruned tree suppresses any transfer whose donor-side copy left no surviving descendant, taking the record of the transfer with it, a suppression that touches about a fifth of all cells on an ordinary run.

A cell is the event, not a label for it. "Ortholog" and "paralog" are readings laid over the event and the published definitions disagree, so ZOMBI2 reports the event and the definition stays yours. For genes to build a species tree from, use the marker table.

The marker table¶

--format markers asks a question about a family, not about a pair: can I put this one in a concatenation and trust the tree that comes out? One row per family that left a surviving copy:

Column	Meaning
`family`	the family id
`genomes` · `copies`	how many extant genomes carry it, and how many copies in total
`single_copy`	every genome that has it has exactly one, so there is no choosing which copy to align
`universal`	every extant genome has it (the criterion a BUSCO-style marker set is built on)
`duplications` · `transfers` · `losses`	the family's own history, dead lineages included
`rf`	Robinson–Foulds distance between the family's true gene tree, each gene read as the genome it sits in, and the species tree restricted to those genomes. Empty where it would mean nothing: several copies in one genome (no one-to-one gene→genome map) or fewer than three genomes (no clade to disagree about)
`congruent`	`rf` is 0, so the family recovers the species tree exactly

The last two are what the table is for. A family can be single-copy and universal and still give the wrong tree: a duplication followed by loss of the other copy in each descendant, or a transfer that replaced the resident gene. That is hidden paralogy, and in real data it passes every filter you would apply. On a transfer-rich run with replacing transfers, 111 of 299 families came out single-copy and universal, and 106 of those did not recover the species tree.

The yes-or-no columns hold literally yes and no, and congruent is empty exactly where rf is:

# every family that would make a trustworthy marker
zombi2 tools format out/ --format markers
awk -F'\t' 'NR==1 || ($4=="yes" && $5=="yes" && $10=="yes")' out/genomes/markers.tsv

rf counts rooted clades present in one tree and not the other, a clade being each internal node's tip set, sizes 2 to n−1, so singletons and the whole tree are excluded. It is the same count zombi2 tools treedist reports, so the two agree.

recPhyloXML¶

recPhyloXML (Duchemin et al. 2018) is the community format for a gene tree embedded in a species tree: every gene-tree node carries the species branch it sat on and the event that ended it, so a viewer can draw one inside the other. It is normally the output of a reconciliation method; here nothing is reconstructed, so the file is the true history.

The complete gene tree goes inside the complete species tree, because the events the format exists to show are the losses, and a gene that died leaves nothing in the extant tree to hang a <loss> on. Extinct and unsampled species are kept for the same reason: a transfer can arrive from a lineage that later died, and that edge has to land somewhere.

ZOMBI2	recPhyloXML
duplication	`<duplication speciesLocation="n<species>">`
speciation	`<speciation speciesLocation="n<species>">`, the parent species, the branch the gene was on when its species split
loss	`<loss speciesLocation="n<species>">`
gene at an extant tip	`<leaf speciesLocation="n<species>">`
gene at an extinct or unsampled tip	`<leaf …>` as well, since the species tree says which fate that branch had
transfer	the format's own two steps: `<branchingOut speciesLocation="n<donor>">` on the node the copy left from, and `<transferBack destinationSpecies="n<recipient>">` opening the child that arrived

Origination has no tag and needs none: a family founded mid-branch is a gene tree whose root starts there. Branch lengths are left out, as in the format's own reference files; the dated trees are next door in genomes/gene_trees/ and species/species_complete.nwk.

In Python, zombi2.tools.recphylo.recphylo_xml(gene_trees, tree) returns the document as a string. gene_trees is a dict of family id to gene tree: g.gene_trees whole for one file a viewer can draw every family in, or a one-entry slice like {7: g.gene_trees[7]} for that family's own. Then tree is the complete species tree, sp.complete_tree.

The extant-only reconciliation¶

--recphylo extant writes the same history projected onto the extant gene tree and the extant species tree, which is what a reconciliation method is scored against, because that is all it ever sees. --recphylo both writes both scopes.

The projection keeps what is observable. A speciation where the gene followed one daughter becomes a single loss on the other, however many losses really happened inside that clade; it disappears entirely when the abandoned daughter has no surviving descendant. A duplication whose second copy died disappears too.

Transfers are the interesting case. When a copy arrives from a lineage that leaves no survivor, the transfer node vanishes with the donor. If the arriving copy has surviving relatives in the gene tree, the transfer is rewritten to leave from those relatives' shared ancestral branch, the point where the copy rejoins them, and the strongest claim the extant tree can express. If it has none, the family appears to begin where it landed.

	rooted at	what it is for
`recphylo_fam<f>_true.xml`	where the family really originated	the answer key, with the losses that narrowed it, which is what an ancestral gene-content reconstruction is trying to recover
`recphylo_fam<f>_recoverable.xml`	the surviving copies' common ancestor	the ceiling. A family that left no trace above that point cannot be placed higher by any method

Both are written so the gap between them is visible: that gap is the part of the history a perfect method still cannot reach.

family_origins.tsv sits beside them, one row per family, with family, entered_by, branch and losses. entered_by is origination when the family really began on the branch it is rooted at, and transfer when it arrived there from a lineage nobody can see; losses counts the loss leaves in the true file.

Unsampled survivors are treated exactly as extinct lineages: nothing in the data refers to either.