| Title: | Tools to Reshape and Manipulate Dichotomous Keys Used in Taxonomic Identification |
|---|---|
| Description: | Transforms between dichotomous identification keys and a variety of phenotype formats, including feature matrices, and generates interactive dichotomous key wizards. Imports keys from 'FishBase' <https://www.fishbase.se/keys/allkeys.php> and includes PhyloTree Build 17, the human mitochondrial DNA phylogeny (van Oven 2015, <doi:10.1016/j.fsigss.2015.09.155>), as distributed by 'Haplogrep' 3. |
| Authors: | Jeremy Leipzig [aut, cre] (ORCID: <https://orcid.org/0000-0001-7224-9620>) |
| Maintainer: | Jeremy Leipzig <[email protected]> |
| License: | GPL (>= 3) |
| Version: | 0.3.0 |
| Built: | 2026-11-21 04:17:38 UTC |
| Source: | https://github.com/leipzig/dimoose |
Computes the tests of a keyFromAlignment() key for each sequence, for
classify(). The sequences must be aligned to the same coordinates the
key was built with: the same alignment columns or, for a key built with a
reference, an alignment that contains that reference sequence.
alignmentScores(key, alignment, reference = NULL)alignmentScores(key, alignment, reference = NULL)
key |
A key from |
alignment |
As in |
reference |
Sequence to number sites by; by default the reference the key was built with, if any. |
A sequences x features matrix: 1 if the sequence has the state
the first lead of a couplet asks for, 0 if it has another state, and
NA if it has a gap or an ambiguity code there.
The key from Triplehorn & Johnson (2005), Borror and DeLong's
Introduction to the Study of Insects, 7th ed., chapter 5, as a table of
leads (see keyFromLeads()); arachnidaKey() turns it into a dimoose
key. Transcribed by hand from the book.
arachnidaarachnida
A data frame with 22 rows (11 couplets) and 6 columns:
Couplet number.
a for the first lead (1.), b for the second (1'.).
Text of the lead.
Couplet the lead leads to, or "-".
Order the lead ends in, for terminal leads.
Page of the book where that order is treated.
Triplehorn CA, Johnson NF (2005). Borror and DeLong's
Introduction to the Study of Insects, 7th ed. Thomson Brooks/Cole.
Transcription: data-raw/arachnida/ in the package sources.
Measures how often photos are identified correctly, by one of three routes:
benchmarkKey( x, photos, model = NULL, route = NULL, top = 3, test = NULL, split = NULL, seed = 1, dir = NULL, prompt = "a photo of %s.", lineage = NULL, confidence = 0.8, k = 40, tiles = 4, method = "rpart", embeddings = NULL, reps = 1 )benchmarkKey( x, photos, model = NULL, route = NULL, top = 3, test = NULL, split = NULL, seed = 1, dir = NULL, prompt = "a photo of %s.", lineage = NULL, confidence = 0.8, k = 40, tiles = 4, method = "rpart", embeddings = NULL, reps = 1 )
x |
A dimoose key, or (for |
photos |
The photos: the manifest |
model |
|
route |
|
top |
Report the accuracy within this many guesses ( |
test |
Share of each species' observations to test with. By default
0.3 for |
split |
Instead of |
seed |
Seed for the split and for finding sniglets. |
dir |
The folder the manifest's |
prompt, lineage, confidence
|
As in |
k, tiles, method
|
For |
embeddings |
For |
reps |
Number of splits to run. One split of a small sample says
little: six test photos per species move a species' accuracy by 17
points each. With |
"names": the model ranks the key's own species for each photo from
their names alone (suggestSpecies()); nothing is learned from the
photos. With a lineage the answers of identifyPhotos(), which fall
back to the genus, family or order when the species is uncertain, are
scored too.
"key": a key with machine tests (from keyFromSniglets() or
keyFromClusters()) is followed for each photo (scoreImages(),
evaluateKey()).
"build": a photo key is built from the training photos (sniglets
found in them, then keyFromSniglets()) and followed for the test
photos. This is the route to use for "how good a key can be learned
from photos like these".
Training and test photos are divided by observation with
splitByObservation(), never photo by photo.
An object of class dimooseBenchmark, a list with
route, n (test photos), nTrain, observations (test observations);
accuracy: share of test photos identified correctly (top-1);
interval: its 95% Wilson interval (lower, upper), which treats the
test photos as independent draws: read it as the least uncertainty
there is, not all of it;
topK: named vector of the accuracy within 1, ..., top guesses;
perSpecies: for each species its number of test photos, top1 and
(for "names") the accuracy within top guesses. Every species of
the photos (and of the key) has a row; one without test photos has
n = 0 and NA accuracies;
confusion: table of actual (rows) against predicted (columns), with
the same species in the same order on both sides, so that the row sums
are the n of perSpecies and the diagonal holds the correct answers.
Any other answer has a column after them: a leaf that names several
species, and "<none>" when a key gave no answer;
results: one row per test photo with id, observation, actual,
predicted, rank (where the true species came in the ranking) and
correct;
split: "train" or "test" for every photo given;
with reps above 1, all of the above describe the first split, and
there are also repeats (one row per split: rep, seed, n,
nTrain, correct, accuracy), accuracyMean, accuracySd (the
standard deviation between splits) and accuracyRange, and
perSpecies gains top1Mean, the species' accuracy over all splits'
test photos together. The splits share photos, so they are not
independent samples: the spread shows how much one split can mislead,
it is not a standard error;
key (for "build", the key that was built) and identify (for
"names" with a lineage: the identifyPhotos() answers with
correct, and the shares that were confident and correct).
inatPhotos(), evaluateKey(), looKey()
## Not run: key <- montanaKey() photos <- inatPhotos(key, n = 20, dir = file.path(tempdir(), "inat")) model <- visionModel(name = "bioclip-2") benchmarkKey(key, photos, model) # species ranked from their names benchmarkKey(model, photos) # a key built from 70% of the observations ## End(Not run)## Not run: key <- montanaKey() photos <- inatPhotos(key, n = 20, dir = file.path(tempdir(), "inat")) model <- visionModel(name = "bioclip-2") benchmarkKey(key, photos, model) # species ranked from their names benchmarkKey(model, photos) # a key built from 70% of the observations ## End(Not run)
Looks up the scientific name of every result in the key in the
GBIF Backbone Taxonomy
and reports names that are misspelled, that are synonyms of a name GBIF
now accepts, or that GBIF does not know. A result written as
"Common name (Genus species)" is checked by the name in parentheses;
other results are checked when they look like a scientific name.
checkNames(key, kingdom = NULL)checkNames(key, kingdom = NULL)
key |
A dimoose key. |
kingdom |
Optional kingdom ("Plantae", "Animalia", "Fungi", ...), which helps GBIF tell apart names that are used in more than one kingdom. |
This needs an internet connection: one request to GBIF per name, and one more for each synonym.
A data frame with one row per name checked: Taxon (as in the
key), name (the scientific name checked), status ("ok",
"misspelled", "synonym", "not found", or "unchecked" when GBIF
could not be reached), gbif (the name GBIF matched), accepted (the
name GBIF accepts for it), genus, family and order (from GBIF's
classification, used by identifyPhotos()) and note (what to fix, in
words).
keyFromSpreadsheet(checkNames = TRUE), which runs this check
and reports problems by spreadsheet row.
checkNames(montanaKey(), kingdom = "Plantae")checkNames(montanaKey(), kingdom = "Plantae")
Walks each row of scores through key: at every couplet the lead whose
test (Feature, Test, Threshold) passes is taken. When no lead or
more than one passes (for instance on a missing score), the result is NA.
classify(key, scores)classify(key, scores)
key |
A dimoose key with |
scores |
A numeric matrix, one row per object, with a column for every
feature id the key tests (see |
A data frame with id (row names of scores), result and
path (the leads taken, e.g. "1a 3b").
Makes the model function suggestSnigletNames() needs from the
Anthropic Messages API:
each call sends one sniglet's crops and the prompt, and returns the text of
the reply. Every call is billed to the API key's account; one call is made
per sniglet, plus one for every name that has to be asked for again.
claudeNamer( model = "claude-opus-5-5", apiKey = Sys.getenv("ANTHROPIC_API_KEY"), maxTokens = 16000, effort = NULL, retries = 3, timeout = 120, url = "https://api.anthropic.com/v1/messages", fallback = FALSE )claudeNamer( model = "claude-opus-5-5", apiKey = Sys.getenv("ANTHROPIC_API_KEY"), maxTokens = 16000, effort = NULL, retries = 3, timeout = 120, url = "https://api.anthropic.com/v1/messages", fallback = FALSE )
model |
The model to ask. It has to accept images. |
apiKey |
The API key. By default the |
maxTokens |
The most tokens a reply may use. Current models think before they answer and the thinking counts, so leave room for it; only the tokens actually used are billed. |
effort |
Optional effort level ( |
retries |
How many times to try again after a rate limit or a server error. |
timeout |
Seconds to wait for a reply. |
url |
The API endpoint. |
fallback |
|
A rate limit (HTTP 429), an overloaded service (529) or a server or
connection error is tried again up to retries times, waiting as long as
the API asks. Any other error stops that one call with the API's message,
which suggestSnigletNames() reports for that sniglet before carrying on
with the next. So does a reply the model declined to give (stop reason
"refusal") and one that ran out of tokens ("max_tokens").
A function function(images, prompt) returning the reply's text.
The API takes JPEG, PNG, GIF and WebP images of at most 5 MB each (as base64; some platforms allow more), and reads small images poorly. With the magick package installed, crops larger than 2000 pixels or 5 MB are scaled down before they are sent and crops smaller than 200 pixels are enlarged. Without it they are sent as they are, and a crop over 5 MB stops that call with a message. A call with more than 100 images is refused, and with many large crops each is scaled down further so that the request stays under the API's 32 MB.
## Not run: # needs an API key in ANTHROPIC_API_KEY; the calls are billed to it key <- patchExemplars(vision, key, images) suggestSnigletNames(key, claudeNamer(), file = "names.csv") # ... review names.csv ... key <- renameSniglets(key, "names.csv") ## End(Not run)## Not run: # needs an API key in ANTHROPIC_API_KEY; the calls are billed to it key <- patchExemplars(vision, key, images) suggestSnigletNames(key, claudeNamer(), file = "names.csv") # ... review names.csv ... key <- renameSniglets(key, "names.csv") ## End(Not run)
An R6 object holding a dichotomous key as a table of the leads on every
path from the root to a terminal taxon. Usually created by
importFishbase() or parseFishbase(). See
https://www.fishbase.se/keys/allkeys.php for FishBase keys.
descKey title or description (read-only).
dfData frame with one row per lead (or per term, if terms were
separated) per terminal taxon, with columns Statement, Choice,
Character, Taxon, and pSt/pCh for the parent lead.
metaKey/value data frame of metadata (citation, image URLs, ...).
taxaData frame of terminal taxa and their resolved names.
leadsData frame with one row per lead: Statement, Choice,
Character, Next (next couplet, or "-" if terminal) and Taxon,
plus optional Prev, Image, ImageLink, TaxonUrl, Question,
Feature, Test and Threshold. Derived from df when the object
was created without one.
featuresData frame of features a machine can compute, one per
row (see featureTable()), or NULL for keys meant only for people.
dimoose$new()Create a new dimoose object.
dimoose$new(df, desc = NA, meta = NA, taxa = NA, leads = NULL, features = NULL)
dfData frame of leads (see the df field).
descKey title or description.
metaKey/value metadata data frame.
taxaData frame of terminal taxa.
leadsOptional data frame of leads (see the leads field).
featuresOptional data frame of machine-readable features (see
featureTable()).
dimoose$print()Print a short overview of the key.
dimoose$print(...)
...Unused.
dimoose$summary()Counts describing the key and how quickly it reaches a
result (see keyQuality()).
dimoose$summary(weights = NULL)
weightsOptional named numeric vector of how common each taxon
is, as for keyQuality(); by default all taxa count the same.
A one-row data frame: title, couplets, leads, taxa,
depth (longest path in steps), problems (see validate()),
meanLeadSteps and meanTaxonSteps (steps to a result, averaged
over terminal leads and over taxa), expectedSteps (averaged over
taxa by weights), bound (the expected steps of the best key
there could be to these taxa, with couplets no larger than this
key's largest) and efficiency (bound / expectedSteps, 1 for a
key that could not be shorter).
dimoose$validate()Check the key: leads to missing couplets, terminal leads without a taxon, couplets with fewer than two leads, couplets that cannot be reached from the first one, and (for keys with features) leads whose machine test is missing or inconsistent.
dimoose$validate()
A character vector of problems, invisibly if there are none.
dimoose$toDataTree()Convert the key to a data.tree::Node: couplets are
internal nodes and taxa leaves; each child node carries the lead
label and the character text of the lead that leads to it. A
couplet reached from several leads is attached under the first.
dimoose$toDataTree()
dimoose$toNewick()Write the key's topology in Newick format, with couplets as internal node labels and taxa as leaves (labels quoted when needed). A couplet reached from several leads is placed under the first.
dimoose$toNewick()
A single string ending in ;.
dimoose$clone()The objects of this class are cloneable with this method.
dimoose$clone(deep = FALSE)
deepWhether to make a deep clone.
A sniglet is a visual feature that a model found and that has no name of
its own, so dimoose coins one for it ("dulmzil"). This function runs
k-means on every patch embedding of every image and keeps the unit length
centroids as the definitions of k sniglets. An image "has" a sniglet
when one of its patches is close to that centroid (see snigletScores()).
Once you have looked at what a sniglet picks out, give it a real name with
renameSniglets().
discoverSniglets(patches, k = 40, seed = 1, nstart = 10, exemplars = 4) discoverTerms(...)discoverSniglets(patches, k = 40, seed = 1, nstart = 10, exemplars = 4) discoverTerms(...)
patches |
A numeric array |
k |
Number of sniglets. |
seed |
Seed for k-means and for |
nstart |
Random starts for |
exemplars |
Number of example regions to record per sniglet, each from a different image. |
... |
Passed to |
discoverTerms(), termScores() and keyFromTerms() are the
earlier names of discoverSniglets(), snigletScores() and
keyFromSniglets(), and still work.
An object of class dimooseSniglets: sniglets (data frame with
id, sniglet (the coined word), name (what people see; the coined
word until renamed) and definition), centroids (k x d, rows named
by coined word), exemplars (data frame sniglet, image, patch,
score: the best-matching region of each of the best-matching images),
best (for every image, the region that matches each sniglet best) and
tiles (the tiling of the regions, see embedImages()).
set.seed(1) patches <- array(rnorm(5 * 9 * 8), c(5, 9, 8)) norms <- sqrt(apply(patches^2, c(1, 2), sum)) patches <- sweep(patches, c(1, 2), norms, "/") dimnames(patches) <- list(letters[1:5], NULL, NULL) discoverSniglets(patches, k = 3)$snigletsset.seed(1) patches <- array(rnorm(5 * 9 * 8), c(5, 9, 8)) norms <- sqrt(apply(patches^2, c(1, 2), sum)) patches <- sweep(patches, c(1, 2), norms, "/") dimnames(patches) <- list(letters[1:5], NULL, NULL) discoverSniglets(patches, k = 3)$sniglets
Returns one embedding per image and, for discoverSniglets(), embeddings
of local regions of each image.
embedImages( model, paths, region = NULL, patches = TRUE, tiles = 4, minDetail = 8, minFill = 0.5 )embedImages( model, paths, region = NULL, patches = TRUE, tiles = 4, minDetail = 8, minFill = 0.5 )
model |
|
paths |
Image files, or an |
region |
Optional crop |
patches |
If |
tiles |
Size of the grid of tiles: |
minDetail |
Tiles with less detail than this (standard deviation of the grey levels, 0 to 255) are treated as empty background and match nothing. |
minFill |
In a photo with a plain background (one colour around most of its border), tiles in which the subject covers less than this share are treated as empty too, so that features are about the subject and not about its backdrop. It has no effect on photos without a plain background. |
By default the regions are tiles: overlapping windows cut out of the
image and embedded one by one, as if each were a whole image. A tile's
embedding describes only what is inside it, so tiles find parts and
surfaces that different images have in common (a stem end, a dimpled
peel). Tiles that are mostly plain background are left out. tiles = 0 uses the model's own patch tokens instead, which is
faster (one pass per image) and is what fordera does, but in the last
layer of a CLIP model each token carries much of the whole image, so the
features found are less local.
A list: image (images x d matrix, rows named by image id: the
file name without extension, or the id of an imageSet()) and
patches (images x regions x d array, or NULL), which remembers the
tiling in its tiles attribute.
Embed texts
embedTexts(model, texts)embedTexts(model, texts)
model |
|
texts |
Character vector, without missing values. |
A texts x d matrix with the texts as row names (no rows for no
texts).
Accuracy of a key followed by machine
evaluateKey(key, scores, labels, groups = NULL)evaluateKey(key, scores, labels, groups = NULL)
key |
A dimoose key with |
scores |
A numeric matrix, one row per object, with a column for every
feature id the key tests (see |
labels |
True label of each row of |
groups |
Optional named vector mapping labels to coarser groups (e.g. model years to generations) for a second accuracy figure. |
A list: results (one row per object with actual, predicted,
correct, groupCorrect), accuracy, groupAccuracy and confusion.
Ready-made dimoose keys built from the datasets that come with dimoose. All
five are ordinary dimoose objects: they share the same fields (leads,
df, meta, taxa, desc) and methods (print(), summary(),
validate(), toDataTree(), toNewick()) and all work with
exportWizard().
sharkKey() arachnidaKey() vibrioKey(variable = c("both", "missing")) montanaKey()sharkKey() arachnidaKey() vibrioKey(variable = c("both", "missing")) montanaKey()
variable |
For |
sharkKey(): FishBase key 1, Key to the families of sharks in the
Western Central Pacific (Compagno 1998), from saved FishBase pages in
inst/extdata/fishbase/; 23 couplets, 24 families.
arachnidaKey(): the Key to the Orders of Arachnida from Triplehorn &
Johnson (2005), the arachnida dataset; 11 couplets, 11 orders
(Schizomida is reached from two couplets).
vibrioKey(): a key generated from the vibrio consensus matrix of
Noguerola & Blanch (2008) with keyFromMatrix() (the matrix itself is
vibrioMatrix()). Tests scored + or (+) count as positive and - or
(-) as negative. With variable = "both" (the default) a species
that is variable for a test (v, d) is keyed out down both leads of
every couplet that uses the test, and so is a species the test was not
determined for (ND): 92 couplets, 74 species, 14 of them reached at
two or three results. With variable = "missing" those results are
treated as missing and the key is an rpart classification tree
(keyFromRpart()), which places such a species on one side of the test
with surrogate splits: 73 couplets, each species at one result, but a
strain showing the species' other result is keyed out wrongly.
montanaKey(): the native broadleaf trees of Montana, written for
dimoose (the montana dataset); 10 couplets, 11 species, with photos
from Wikimedia Commons. It is built with keyFromSpreadsheet() from
inst/extdata/montana/montana-key.csv, a template for writing your own.
phylotreeKey(): PhyloTree Build 17.
A dimoose object.
Compagno LJV (1998). General remarks. In Carpenter KE, Niem VH (eds.), FAO species identification guide for fishery purposes. The living marine resources of the Western Central Pacific, vol. 2, pp. 1196-1207. FAO, Rome. FishBase key 1: https://www.fishbase.se/keys/description.php?keycode=1.
Triplehorn CA, Johnson NF (2005). Borror and DeLong's Introduction to the Study of Insects, 7th ed., chapter 5. Thomson Brooks/Cole. ISBN 0-03-096835-6.
Noguerola I, Blanch AR (2008). Identification of Vibrio spp. with a set of dichotomous keys. Journal of Applied Microbiology 105, 175-185. doi:10.1111/j.1365-2672.2008.03730.x
sharks <- sharkKey() sharks summary(arachnidaKey()) vibrioKey()$validate() summary(vibrioKey(variable = "missing")) ## Not run: exportWizard(sharkKey(), "sharks.html") ## End(Not run)sharks <- sharkKey() sharks summary(arachnidaKey()) vibrioKey()$validate() summary(vibrioKey(variable = "missing")) ## Not run: exportWizard(sharkKey(), "sharks.html") ## End(Not run)
Photo identification as suggestSpecies() and identifyPhotos() do it has
two halves: the names of the key's results are embedded as texts, and the
photo is embedded as an image and compared with them. The text half can be
done ahead of time, here; a web page then needs only the image encoder to
identify a photo on the visitor's own device, with no server and no
upload. This function writes the files such a page loads.
exportBrowserModel( key, model, dir, encoder = NULL, prompt = "a photo of %s.", textEmb = NULL )exportBrowserModel( key, model, dir, encoder = NULL, prompt = "a photo of %s.", textEmb = NULL )
key |
A key whose results are the labels. |
model |
The |
dir |
Folder to write to; created if needed. |
encoder |
The image encoder for the page: |
prompt |
The text each name is put into, with |
textEmb |
Optional text embeddings made earlier, a matrix whose row
names are the prompts (from |
The path of model.json, invisibly.
dir
model.json, the manifest (UTF-8 JSON):
format: "dimoose-browser-model"; version: 1.
key: desc (the key's description) and taxa (the number of labels).
model: arch, pretrained, dim (embedding dimensions), library.
image_encoder: file (a file name in dir, or null), url (a web
address, or null; one of the two is set), bytes, precision
("int8", "fp16", "fp32" or null if unknown), input (the input's
name, "pixel_values"), input_type ("float32"), input_shape
([1, 3, S, S]), output (the output to read, "image_embeds"),
normalized (false: divide the output by its length).
preprocess, how to turn a photo into the input: resize
("shortest": scale so the shorter side is resize_to pixels, keeping
the shape), interpolation ("bicubic"), crop ("center": cut the
middle image_size x image_size square), rescale (multiply the 0-255
values by this, 1/255), mean and std (per channel: subtract, then
divide), layout ("NCHW"), channels ("RGB").
text_embeddings: file ("text_embeddings.bin"), dtype
("float32"), byte_order ("little"), shape ([labels, dim]),
order ("row-major": label by label), normalized (true), prompt
(the template).
labels: one object per row of the text embeddings, in order: taxon
(the result as the key's leads write it, which is what to look up in the
key), name (the name put into the prompt: the scientific name when the
result has one) and prompt (the text that was embedded).
logit_scale: 100.
text_embeddings.bin: labels x dim 32-bit little-endian floats and
nothing else, row by row; in JavaScript,
new Float32Array(await (await fetch(url)).arrayBuffer()).
image_encoder.onnx: the image encoder, unless the manifest gives a
url instead.
Prepare the photo as preprocess says, run the image encoder (for example
with onnxruntime-web), divide image_embeds by its length to get v, and
take the dot product of v with each row of the text embeddings. The
probability of each label is the softmax of logit_scale times those dot
products, which is what suggestSpecies() computes.
The image encoder is the download that matters. Use an "int8" encoder
for the web: 96 MB for CLIP ViT-B/32 and 90 MB for BioCLIP as
exportOnnxModel() writes them, against about 350 MB unquantised, with
the same answers on dimoose's example photos (BioCLIP 2, a larger model,
is 310 MB even so). Where that is still too much, "mobileclip-s0" at
"fp16" is 23 MB, but it is a general model that does not know species
names as BioCLIP does. The text embeddings are tiny
(2 KB per label), and are best made with the "fp32" text encoder:
onnxModel(name, precision = "int8", textPrecision = "fp32").
The "int8", "fp16" and "fp32" encoders that exportOnnxModel()
writes for CLIP ViT-B/32 and BioCLIP were run in onnxruntime-web 1.31.1 on
its WebAssembly backend (under Node.js) from folders this function wrote,
with the 14 Montana example photos prepared as preprocess says. All
three load and run there. Against the same file run in Python the lowest
cosine of the embeddings was 0.9996 for "int8" and 1.0000 for "fp16"
and "fp32", and the most likely label was the same for every photo. The
"int8" files hold 8-bit weights that are turned back into floats when
the model loads, so they use no integer arithmetic a runtime might lack.
The WebGPU backend was not tested.
Ready-made _quantized files from the Hub (encoder = "hub" with an
"int8" model) are quantised differently, with integer arithmetic. That
of Xenova/clip-vit-base-patch32 had a lowest cosine of 0.85 with the
"fp32" embeddings on the same backend and gave another most likely label
for 5 of the 14 photos; its "fp16" file had 1.0000. Point a page at the
"fp16" file, or copy in an "int8" encoder from exportOnnxModel().
How the photo is resized matters more than the encoder's precision.
preprocess$interpolation names the filter the model was trained with
(bicubic, with smoothing when a photo is made smaller, as the Python
library Pillow does it). With BioCLIP on the Montana photos, resizing with
a bilinear filter instead gave embeddings with a cosine of 0.93 to 0.98
(0.97 on average) with the right ones, and nearest-neighbour 0.84 to 0.95.
onnxModel(), exportOnnxModel(), suggestSpecies().
## Not run: model <- onnxModel("bioclip", precision = "int8", textPrecision = "fp32") exportBrowserModel(montanaKey(), model, "site/model") list.files("site/model") # image_encoder.onnx, model.json, text_embeddings.bin ## End(Not run)## Not run: model <- onnxModel("bioclip", precision = "int8", textPrecision = "fp32") exportBrowserModel(montanaKey(), model, "site/model") list.files("site/model") # image_encoder.onnx, model.json, text_embeddings.bin ## End(Not run)
Runs dimoose's export_onnx.py (in the package's python folder) to write
a model as ONNX image and text encoders that onnxModel() runs without
PyTorch. The conversion itself needs PyTorch and open_clip, but only once:
the folder it writes can be copied to other computers or uploaded to a
Hugging Face repository and used from there.
exportOnnxModel( name = "bioclip", pretrained = "openai", weights = NULL, dir = NULL, quantize = c("int8", "fp16"), python = NULL, run = NULL )exportOnnxModel( name = "bioclip", pretrained = "openai", weights = NULL, dir = NULL, quantize = c("int8", "fp16"), python = NULL, run = NULL )
name |
The model, as for |
pretrained |
open_clip's tag for the weights of an architecture. |
weights |
Local checkpoint file instead of a download. |
dir |
Folder to write to. By default the model's folder in
|
quantize |
Which smaller copies to write as well: |
python |
The Python to run the script with; it needs the packages in
the package's |
run |
For tests: a function |
Each file is checked against the PyTorch model as it is written; the script prints the files' sizes and how close their embeddings are.
The folder, invisibly.
dimoose_onnx.json: architecture, image size, patch size, embedding
dimensions, preprocessing (resize, interpolation, crop, rescale,
mean, std), the file names and sizes of each precision, and the names
of the encoders' inputs and outputs.
image_encoder.onnx: input pixel_values (n x 3 x S x S, float32);
outputs image_embeds (n x d, not normalised) and patch_embeds
(n x g*g x d, the projected patch tokens).
text_encoder.onnx: input input_ids (n x 77, int64); output
text_embeds (n x d, not normalised).
tokenizer.json: the tokenizer in the Hugging Face tokenizers format,
which transformers.js also reads.
with quantize: image_encoder_int8.onnx, text_encoder_int8.onnx
(weights as 8-bit integers, about a quarter of the size) and
image_encoder_fp16.onnx, text_encoder_fp16.onnx (16-bit floats, half
the size, same embeddings).
## Not run: exportOnnxModel("bioclip") model <- onnxModel("bioclip", precision = "int8") ## End(Not run)## Not run: exportOnnxModel("bioclip") model <- onnxModel("bioclip", precision = "int8") ## End(Not run)
Writes a dimoose key as a single self-contained HTML page. With JavaScript
on, the page steps through the key one couplet at a time, keeps a trail of
the choices made (each one clickable to go back to it) and supports the
browser's Back button. Without JavaScript, the same page is the full key
with every lead hyperlinked to its next couplet or taxon. Each taxon gets
a result section listing the characters on its path, and a map of the
key shows where the user is (see map).
exportWizard( key = NULL, file = NULL, displayLinks = TRUE, order = "original", title = NULL, map = TRUE, staticPaths = NULL, matrix = NULL, characters = NULL, taxon = "Taxon", offline = FALSE, photoModel = NULL, photoRuntime = NULL )exportWizard( key = NULL, file = NULL, displayLinks = TRUE, order = "original", title = NULL, map = TRUE, staticPaths = NULL, matrix = NULL, characters = NULL, taxon = "Taxon", offline = FALSE, photoModel = NULL, photoRuntime = NULL )
key |
A dimoose object, e.g. from |
file |
Path of the HTML file to write. If |
displayLinks |
If |
order |
Deprecated and ignored. Couplets are always exported in the
key's own order. |
title |
Page title. Defaults to the key's description. |
map |
If |
staticPaths |
If |
matrix |
Optional character matrix: a data frame with one row per
taxon, the taxon's name in column |
characters |
Optional data frame describing the characters of
|
taxon |
Name of the column of |
offline |
If |
photoModel |
Optional: a folder written by |
photoRuntime |
Optional web address of a folder holding the |
The key is checked first: leads pointing at missing couplets, terminal leads without a taxon, couplets with fewer than two leads and couplets that cannot be reached from the first one are reported as warnings and marked in the page.
The path to file, invisibly, or the HTML as a character string if
file is NULL. With offline = TRUE or a photoModel folder the
paths of the other files written are in attribute "files".
Every couplet has an "I'm not sure" button (keys u or ?). It follows
all leads of the couplet at once. The page then asks the questions of one
branch at a time, lists the branches still open (click one to answer it
next), and counts and marks on the map the taxa possible in any of them.
Each answer narrows its own branch. If the reader stays unsure to the end,
the page lists every result that is left. Back undoes one step at a time.
The address of the page records the open branches (#open=c-3,c-9), so a
reload or a shared link returns to the same place; the trail of choices
comes back too when the browser restores the page's history entry.
With matrix, the page gains a second way in: a list of the characters of
the matrix, to answer in any order. Picking a state removes the taxa that
do not have it. A taxon with several states ("a/b") fits any of them, and
a taxon whose state is unknown (NA) fits all. The characters that still
separate the remaining taxa are listed first: cheapest first when
characters gives a Cost, then those that leave the fewest taxa on
average. Taxa of the matrix are matched to the key's taxa by name (ignoring
case), so the remaining taxa link to their result sections and show on the
map, and the page offers to continue step by step from the couplet they all
share. Without JavaScript the matrix is a plain table. The two ways are
otherwise independent: answers given step by step do not filter the
character list. With key = NULL the page is the character list alone.
The page itself needs no connection once it is open, but a browser will not
reopen a page it cannot fetch. offline = TRUE writes four files next to
file, named after it: a web app manifest (<name>.webmanifest), a
service worker (<name>-sw.js) and two icons (<name>-icon-192.png,
<name>-icon-512.png; the package logo if the rsvg package is installed,
otherwise a simple drawn icon). The worker stores the page, the manifest,
the icons and the figures the page loads from other sites, so that the
page can be installed on a phone ("Add to Home Screen") and opened without
signal. Upload all five files to the same folder. Limits:
Browsers run service workers only on https:// or http://localhost. A
page opened from disk (file://) works as before but is not installed
and stores nothing.
The page must be opened once with a connection before it works without.
Figures on other sites are fetched when the worker installs and again
when first shown; any that fail are skipped. A site that does not allow
cross-origin requests gives an opaque copy, which browsers count as
several megabytes each against the storage they allow, so a key with
many such figures may not store them all. Figures embedded in the key
(see imageMeta()) are part of the page and always available. Full-size
figures and taxon pages that open in a new tab are not stored.
The worker only controls the page it was written for (its scope is the
page's own address), so several keys can share a folder. Serve the page
under its file name; a server that drops the .html from addresses puts
the page outside the worker's scope.
photoModel adds an "Identify from a photo" button. The reader takes or
picks a photo; the page loads the model and
onnxruntime-web (the program
that runs it, version 1.31.1, about 15 MB, from the jsDelivr CDN unless
photoRuntime says otherwise), prepares the picture as the model's
model.json says, runs the image encoder in the browser and compares the
result with the text embeddings of the key's taxa. The photo never leaves
the reader's device. The page then marks the likely taxa on the map, lists
the five best with their scores, each linked to its result, and offers to
narrow the key to them: the leads that reach none of them are greyed and
the count of remaining taxa follows, but every lead still works. The key
stays the authority; the photo is a hint.
photoModel is a folder written by exportBrowserModel() for the same
key, or the web address of one. The model is not put into the page. A
folder is copied next to file as <name>_model/ (so file is needed);
upload it with the page. A web address is used as it is; a model on
another site must allow cross-origin requests. What to expect:
Nothing is downloaded until the reader has chosen a photo, been told the size and agreed. An int8 BioCLIP encoder is 90 MB; with the runtime that is about 105 MB on first use. The page shows the progress.
With offline = TRUE the service worker keeps the model and the runtime
once they have been fetched, so photos can be identified without a
connection afterwards. That is the same 105 MB or so of storage on the
reader's device, and the first use still needs a connection.
It needs a browser with WebAssembly, and memory: several times the size
of the encoder while it loads. If the browser cannot run the model, runs
out of memory or cannot fetch it, the page says so and the key works as
before. A page opened from disk (file://) cannot load the model.
Only the WebAssembly backend is used, on one thread unless the page is served cross-origin isolated. That suits int8 encoders, which WebGPU cannot run.
The scores compare the key's taxa with each other and add up to 100%,
so a photo of something that is not in the key still gets a confident
best match. If model.json has a min_similarity, a best match whose
cosine similarity is under it is shown as "Not much like any taxon of
this key" without a percentage.
A Note column in the key's leads, as pruneKey() writes when it
collapses a couplet (the character of the couplet that is no longer asked),
is shown under its lead as "Also:" and in the path of each result. A
pruned_note in the key's meta (the same for a collapsed first couplet) is
shown under the title.
The page defines window.dimoose so that another script, for example one
that identifies a photo in the browser, can work with the page without
knowing how it is built:
dimoose.taxa: array of { id, name }, one per result.
dimoose.highlight(scores, opts): scores is an object mapping taxon
names or ids to numbers, or an array of { taxon, score }; null
clears. Marks those taxa on the map and in every list of candidates and
lists the five best above the key, headed by opts.label (default
"Suggested"). Scores all within 0 to 1 are shown as percentages. Returns
the ids it marked, best first.
dimoose.goToTaxon(nameOrId): shows that result; false if unknown.
dimoose.narrow(namesOrIds): narrows the key to these taxa (see
'Identifying from a photo'); null undoes it. Returns the ids kept.
dimoose.state(): { shown, open, remaining, picks, narrowed }, the
section shown, the open branches, the ids of the taxa still possible, the
states picked in the character list and the ids the key is narrowed to
(or null).
dimoose.photo (only with photoModel): identify(file), a promise of
the ranked taxa, and the pure steps preprocess() and score().
dimoose.onChange(fn): calls fn(state) after every step.
keyFromMatrix() to build a key from a character matrix.
leads <- data.frame( Statement = c("1", "1", "2", "2"), Choice = c("a", "b", "a", "b"), Character = c("Wings present", "Wings absent", "Two wings", "Four wings"), Next = c("2", "-", "-", "-"), Taxon = c("", "Silverfish", "Fly", "Bee") ) key <- keyFromLeads(leads, "A tiny key") html <- exportWizard(key) substr(html, 1, 15) # With a character matrix the reader can also pick characters in any order insects <- data.frame( Taxon = c("Silverfish", "Fly", "Bee", "Ant"), Wings = c("absent", "present", "present", "present/absent"), Sting = c("absent", "absent", "present", NA) ) characters <- data.frame(Character = c("Wings", "Sting"), Cost = c(1, 3), Needs = c("", "hand lens")) html <- exportWizard(key, matrix = insects, characters = characters) # A matrix alone makes a page with only the character list html <- exportWizard(NULL, matrix = insects, title = "Four insects") # An installable copy that opens without a connection once it is served # over https: the page, a manifest, a service worker and two icons dir <- tempfile("key") dir.create(dir) out <- exportWizard(key, file.path(dir, "insects.html"), offline = TRUE) list.files(dir) unlink(dir, recursive = TRUE)leads <- data.frame( Statement = c("1", "1", "2", "2"), Choice = c("a", "b", "a", "b"), Character = c("Wings present", "Wings absent", "Two wings", "Four wings"), Next = c("2", "-", "-", "-"), Taxon = c("", "Silverfish", "Fly", "Bee") ) key <- keyFromLeads(leads, "A tiny key") html <- exportWizard(key) substr(html, 1, 15) # With a character matrix the reader can also pick characters in any order insects <- data.frame( Taxon = c("Silverfish", "Fly", "Bee", "Ant"), Wings = c("absent", "present", "present", "present/absent"), Sting = c("absent", "absent", "present", NA) ) characters <- data.frame(Character = c("Wings", "Sting"), Cost = c(1, 3), Needs = c("", "hand lens")) html <- exportWizard(key, matrix = insects, characters = characters) # A matrix alone makes a page with only the character list html <- exportWizard(NULL, matrix = insects, title = "Four insects") # An installable copy that opens without a connection once it is served # over https: the page, a manifest, a service worker and two icons dir <- tempfile("key") dir.create(dir) out <- exportWizard(key, file.path(dir, "insects.html"), offline = TRUE) list.files(dir) unlink(dir, recursive = TRUE)
Model-free: give it the embeddings and it applies each feature's rule.
centroid_patch_max features need patches; clip_text_pair features
need image and textEmb (rows named by prompt). For external
features supply the scores yourself.
featureScores(key, image = NULL, patches = NULL, textEmb = NULL)featureScores(key, image = NULL, patches = NULL, textEmb = NULL)
key |
A dimoose key with a |
image |
|
patches |
|
textEmb |
|
An objects x features matrix with feature ids as column names.
scoreImages() for images-to-scores in one call.
Keys built from images (see keyFromTerms() and keyFromClusters()) carry
a features table that says how a machine computes the score each lead
tests. classify() walks a key over a matrix of such scores.
featureTable( id, kind, label = id, prompt = NA, negative_prompt = NA, region = NA, embedding = NULL, exemplars = NULL, definition = NA )featureTable( id, kind, label = id, prompt = NA, negative_prompt = NA, region = NA, embedding = NULL, exemplars = NULL, definition = NA )
id |
Feature ids, e.g. |
kind |
|
label |
Name shown to people; defaults to |
prompt, negative_prompt
|
Texts for |
region |
Crop as |
embedding |
List of numeric vectors (one per feature), or |
exemplars |
List of data frames with columns |
definition |
Optional sentence saying what the feature is, shown in
the wizard's glossary (see |
A data frame with the columns above; embedding and exemplars
are list columns.
featureTable("sniglet:x", "centroid_patch_max", embedding = list(c(1, 0, 0)))featureTable("sniglet:x", "centroid_patch_max", embedding = list(c(1, 0, 0)))
Fits a classification tree (or tree ensemble) with the taxon column as the response and every other column as a feature.
generateTree( data, target_col, method = c("rpart", "randomForest", "gbm"), params = list() )generateTree( data, target_col, method = c("rpart", "randomForest", "gbm"), params = list() )
data |
A data frame containing the feature matrix. |
target_col |
Name of the target/response column. |
method |
|
params |
Optional list of arguments passed to the fitting function, overriding the defaults. |
The rpart defaults are set for identification data, which usually has one
row per taxon: rpart.control(minsplit = 2, minbucket = 1, cp = 0), so the
tree keeps splitting until every taxon is separated. (rpart's own defaults,
minsplit = 20, minbucket = 7, cp = 0.01, make no split at all on such a
matrix.) With many specimens per taxon pass your own control in params
to get a smaller tree, or just the settings to change, e.g.
params = list(cp = 0.01) or params = list(maxdepth = 3). keyFromRpart() turns the tree into a key;
keyFromMatrix() goes from matrix to key in one step and handles taxa that
are variable for a character.
A fitted rpart, randomForest or gbm object. For the last two
the call is the call to generateTree().
features <- data.frame( family = factor(c("A", "B", "C")), fin = factor(c("x", "y", "y")), head = factor(c("p", "p", "q")) ) fit <- generateTree(features, "family") fit keyFromRpart(fit)$leadsfeatures <- data.frame( family = factor(c("A", "B", "C")), fin = factor(c("x", "y", "y")), head = factor(c("p", "p", "q")) ) fit <- generateTree(features, "family") fit keyFromRpart(fit)$leads
Follows every haplogroup's path from the root of the tree and records, for
each requested site, whether the haplogroup's haplotype differs from the
root (RSRS) there. Back-mutations are applied as Haplogrep defines them
(they undo the latest change at that site), so a site gained and later
reverted on a lineage is not carried below the reversion. Insertions are
tracked as their own sites ("573.X", "2491.1"), separate from
substitutions at the same position; a deletion counts as carried.
haplotypeMatrix( haplogroups, sites = NULL, tree = dimoose::phylotree17, mutations = dimoose::phylotree17_mutations )haplotypeMatrix( haplogroups, sites = NULL, tree = dimoose::phylotree17, mutations = dimoose::phylotree17_mutations )
haplogroups |
Haplogroup names, one per row of the result; repeats are allowed (e.g. one per sample). |
sites |
Sites to report: positions as character ( |
tree, mutations
|
The tree and its mutations; default phylotree17 and
phylotree17_mutations. |
An integer matrix of 0/1, one row per entry of haplogroups (row
names are the haplogroups) and one column per site.
recurrentPositions(), macroHaplogroup()
haplotypeMatrix(c("H2a2a1", "U5b", "L0a1b"), sites = c("152", "16311", "16278"))haplotypeMatrix(c("H2a2a1", "U5b", "L0a1b"), sites = c("152", "16311", "16278"))
Like suggestSpecies(), compares each photo with the key's own species,
but gives one answer per photo: the species when it alone is likely
enough, otherwise the genus, family or order whose species together are.
A photo of a cottonwood that the model cannot place between the key's two
cottonwoods is reported as genus Populus rather than as a coin toss
between them.
identifyPhotos( key, model, paths, confidence = 0.8, lineage = NULL, kingdom = NULL, prompt = "a photo of %s." )identifyPhotos( key, model, paths, confidence = 0.8, lineage = NULL, kingdom = NULL, prompt = "a photo of %s." )
key |
A dimoose key. |
model |
|
paths |
Image files, or an |
confidence |
The probability an answer must reach, between 0 and 1. |
lineage |
Optional data frame with columns |
kingdom |
Optional kingdom ("Plantae", "Animalia", "Fungi", ...), which helps GBIF tell apart names that are used in more than one kingdom. |
prompt |
The text each name is put into, with |
The genus, family and order of each species come from GBIF (see
checkNames()), which needs an internet connection, or from a table you
give in lineage. Results with no lineage (for example ones named only by
a common name) count only as themselves.
A data frame with one row per image: image, name (the
scientific name of the answer), rank ("species", "genus",
"family" or "order"), probability, confident (whether
probability reaches confidence; when nothing does, the answer is the
most likely species) and Taxon (the key's result, for species-level
answers).
## Not run: model <- visionModel(name = "bioclip-2") identifyPhotos(montanaKey(), model, "my-tree.jpg", kingdom = "Plantae") ## End(Not run)## Not run: model <- visionModel(name = "bioclip-2") identifyPhotos(montanaKey(), model, "my-tree.jpg", kingdom = "Plantae") ## End(Not run)
Reads image files and returns image_<id> rows for a key's meta, which
exportWizard() embeds in the page (each image once); leads refer to them
by id in their Image column (several separated by ;).
imageMeta(paths, ids = NULL, maxSize = 1200)imageMeta(paths, ids = NULL, maxSize = 1200)
paths |
Image files. |
ids |
Names for the images; default the file names without extension. |
maxSize |
Photos larger than this many pixels on their long side are scaled down to it when the 'magick' package is installed, so keys made from camera photos stay small; without 'magick' they are embedded as they are. |
A key/value data frame.
MIME type of an image file from its first bytes
imageMime(path)imageMime(path)
path |
One file. |
"image/png", "image/jpeg", "image/gif", "image/webp" or NA.
The starting point for automated key generation: one row per photo, with the class (species, model, ...) the photo shows. A key separates the classes, so there must be at least two.
imageSet(paths, label = NULL)imageSet(paths, label = NULL)
paths |
Image files, or a single folder. A folder is searched (with its subfolders) for image files: png, jpg/jpeg, gif, webp, bmp and tif/tiff. |
label |
Class of each image:
|
A data frame with id, path and label. id is the file name
without its extension; when two files share a name, their folder is put
in front (cardinal_IMG_01) so that ids are unique.
imageSet(c("pics/1967.png", "pics/1967_alt.png"), label = function(x) sub("_.*$", "", x)) imageSet(c("pics/cardinal/1.jpg", "pics/robin/1.jpg"), label = c("cardinal", "robin")) # the example photos that ship with dimoose: one folder per fruit pics <- imageSet(system.file("extdata", "pics", package = "dimoose")) table(pics$label)imageSet(c("pics/1967.png", "pics/1967_alt.png"), label = function(x) sub("_.*$", "", x)) imageSet(c("pics/cardinal/1.jpg", "pics/robin/1.jpg"), label = c("cardinal", "robin")) # the example photos that ship with dimoose: one folder per fruit pics <- imageSet(system.file("extdata", "pics", package = "dimoose")) table(pics$label)
Downloads a key from FishBase and converts it into a dimoose object. See https://www.fishbase.se/keys/allkeys.php for the list of keys and their key codes.
importFishbase( keycode, fishbaseUrl = "https://www.fishbase.se/", separateTerms = TRUE, usePhyloService = c("GNA", "GNR", "none", "TNRS"), embedImages = FALSE )importFishbase( keycode, fishbaseUrl = "https://www.fishbase.se/", separateTerms = TRUE, usePhyloService = c("GNA", "GNR", "none", "TNRS"), embedImages = FALSE )
keycode |
Numeric or character FishBase key code. |
fishbaseUrl |
Base URL of the FishBase mirror, with a trailing slash. |
separateTerms |
If |
usePhyloService |
Taxonomic name resolution: |
embedImages |
If |
A dimoose object.
parseFishbase() to build a key from saved HTML pages.
## Not run: sharks <- importFishbase(1) sharks$taxa ## End(Not run)## Not run: sharks <- importFishbase(1) sharks$taxa ## End(Not run)
Collects field photos to test a key with: for each species, n
observations from iNaturalist with an openly
licensed photo, the photo files, and a manifest that says where every photo
came from, so that a benchmark can be cited and run again.
inatPhotos( taxa, n = 20, place = NULL, qualityGrade = "research", licenses = c("cc0", "cc-by", "cc-by-nc"), dir, fetch = benchInatGet, download = benchDownload, size = "medium", pool = 200, strata = 10, idBelow = NULL, seed = 1, perObserver = 1, photosPerObservation = 1, query = list(), pause = 1 )inatPhotos( taxa, n = 20, place = NULL, qualityGrade = "research", licenses = c("cc0", "cc-by", "cc-by-nc"), dir, fetch = benchInatGet, download = benchDownload, size = "medium", pool = 200, strata = 10, idBelow = NULL, seed = 1, perObserver = 1, photosPerObservation = 1, query = list(), pause = 1 )
taxa |
The species: scientific names, results written
|
n |
Observations to sample per species. |
place |
Optional place to restrict the sample to: an iNaturalist place id, or a name to look up (the best match is used and reported). |
qualityGrade |
iNaturalist quality grade. |
licenses |
Photo licences to accept. |
dir |
Folder to put the photos and |
fetch |
Function |
download |
Function |
size |
Photo size: |
pool, strata
|
Size of the pool the sample is drawn from, and the number of slices of the id range it is collected from. |
idBelow |
Only observations with an id below this; see above. |
seed |
Seed for drawing the sample from the pool. |
perObserver |
Most observations to take from one observer per species. |
photosPerObservation |
Most photos to take from one observation. |
query |
Further parameters for the observation search, e.g.
|
pause |
Least number of seconds between requests. |
The manifest, a data frame with one row per photo: taxon (the
label, as given in taxa), scientific_name, taxon_id,
observation_id, photo_id, license, attribution, url,
observation_url, observed_on, file (relative to dir),
quality_grade, place_id, id_below and retrieved; plus path, the
full file name on this computer, which is not written to
manifest.csv. benchmarkKey() takes it as its photos.
The sample depends only on the arguments, not on the day it is drawn:
idBelow fixes the snapshot: only observations with a smaller id are
considered. When it is NULL the newest observation id at the time of
the call is used and recorded in the manifest (id_below); give that
value to draw the same sample again later.
For each species a pool of up to pool observations is collected from
strata equal slices of the id range (the newest observations below
each cut), so that the pool spans the years rather than being last
month's photos. Ids follow upload dates, so each slice is a short
stretch of time and the same stretches are used for every species;
raise strata (one request each) for an evener spread over the
seasons.
From the pool, sorted by id, n observations are drawn with seed,
taking at most perObserver from one observer, so that one person's
many photos of the same tree cannot dominate.
Observations that were deleted or lost their research grade since are
simply no longer returned, so an old sample can shrink; the photo files
already in dir are kept and not downloaded again. A photo that cannot be
downloaded, or that turns out not to be an image, is left out of the
manifest with a warning; it is not replaced by another.
A name is looked up in iNaturalist's taxon search and must match a taxon's name exactly; a looser match (the search also finds "Oenantheae" and "Oenanthe oenanthe" for "Oenanthe") is never used. A name iNaturalist lists as a synonym resolves to the taxon it now belongs to, with a message. The same name can belong to two taxa (the genus Oenanthe is a bird and a plant): then the one whose rank fits the name best and that has the most observations is used, with a warning that lists the candidates and their ids. Give the taxon id instead of the name to choose yourself. The manifest records the id that was used.
Requests are spaced at least pause seconds apart (iNaturalist asks for
about one request a second) and identify themselves with a generic
User-Agent. Only photos whose own licence is in licenses are taken;
the manifest keeps each photo's licence and attribution, which the CC BY
licences require you to show wherever the photos are.
## Not run: photos <- inatPhotos(montanaKey(), n = 20, dir = file.path(tempdir(), "inat")) table(photos$taxon) ## End(Not run)## Not run: photos <- inatPhotos(montanaKey(), n = 20, dir = file.path(tempdir(), "inat")) table(photos$taxon) ## End(Not run)
Random words made of consonant onsets, vowels and codas. They name sniglets: visual features that have no English name (as fordera does).
inventNames(n, seed = 1)inventNames(n, seed = 1)
n |
Number of names. |
seed |
Random seed, so the same call gives the same names. The global random number generator state is left unchanged. |
A character vector of n distinct lower-case words.
inventNames(5)inventNames(5)
Draws a dimoose key top-down as an SVG picture, the way keys are often
drawn on paper: each couplet is a blue box holding its question, its leads
are blue boxes below it, and every lead either continues to the next
couplet or ends in a green box naming the result. A result written as
"Common name (Genus species)" is drawn on two lines with the scientific
name in italics.
keyDiagram(key, file = NULL, images = TRUE, title = NULL)keyDiagram(key, file = NULL, images = TRUE, title = NULL)
key |
A dimoose key. |
file |
Optional file to write: |
images |
If |
title |
Heading drawn above the tree; by default the key's |
Thumbnails are shrunk to 112 pixels when the 'magick' package is installed, which keeps diagrams of keys with large photos small. When 'rgnparser' and its 'gnparser' program are installed, only text that parses as a scientific name is set in italics.
A couplet reached from more than one lead is drawn once, under the first of them; the other leads get a dashed line to it.
The SVG as a single string, invisibly when file is given.
exportWizard() for an interactive version of the key.
keyDiagram(arachnidaKey(), tempfile(fileext = ".svg"))keyDiagram(arachnidaKey(), tempfile(fileext = ".svg"))
One command from aligned sequences to a dichotomous key. A phylogeny
program builds a tree from the alignment, the tree is rooted and made
strictly bifurcating, and every fork becomes a couplet whose two leads are
the alignment sites that tell its two clades apart ("152C 16311T"
against "152T 16311C"). Each sequence is a result.
keyFromAlignment( alignment, tool = c("nj", "bionj", "upgma", "parsimony", "ml", "fasttree", "iqtree"), tree = NULL, outgroup = NULL, reference = NULL, type = c("auto", "DNA", "AA"), model = NULL, maxSites = 3, exe = NULL, args = NULL, desc = "Key generated from a sequence alignment", meta = NULL )keyFromAlignment( alignment, tool = c("nj", "bionj", "upgma", "parsimony", "ml", "fasttree", "iqtree"), tree = NULL, outgroup = NULL, reference = NULL, type = c("auto", "DNA", "AA"), model = NULL, maxSites = 3, exe = NULL, args = NULL, desc = "Key generated from a sequence alignment", meta = NULL )
alignment |
The alignment: a FASTA file, an |
tool |
The program that builds the tree:
|
tree |
A tree you built yourself, instead of |
outgroup |
Name(s) of the sequence(s) to root on. Without it, a tree that is not already rooted is rooted at its midpoint. Sequences can be named in full or by the first word of their FASTA header. |
reference |
Name of a sequence in the alignment to number sites by
(its ungapped positions; columns where it has a gap are numbered
|
type |
|
model |
Substitution model, passed to the tool: the |
maxSites |
Most sites listed in a lead. |
exe |
Path to the FastTree or IQ-TREE program, if it is not on the
|
args |
Extra command-line arguments for FastTree or IQ-TREE. |
desc, meta
|
Title and extra metadata of the key. |
Sites are chosen per fork: a site is diagnostic when the states seen on
one side are never seen on the other. Up to maxSites diagnostic sites
are listed, those with the least missing data first. If a fork has no diagnostic site, the
best-separating one is used and marked (most). If no site separates the
two sides, the sequences below it are split by their sites alone,
whatever the tree says. Sequences that no site tells apart share a result
("A / B"), as do identical sequences. Gaps and ambiguity codes are treated as missing data.
The first site of each couplet is also a machine-readable test, so
classify() can place sequences scored with alignmentScores().
A dimoose key. Its meta records the tool, the rooting, the tree
in Newick format (newick, names quoted where Newick needs it), how many couplets lack a diagnostic site
(weak_couplets) and how many do not follow the tree
(regrouped_couplets).
alignmentScores(), phylotreeKey(), toNewick()
aln <- c(human = "ACGTACGTACGA", chimp = "ACGTACGTATGA", mouse = "ACCTACATGCGA", rat = "ACCTACATGCGT", fish = "TCCTTCGTACCA") if (requireNamespace("ape", quietly = TRUE)) { key <- keyFromAlignment(aln, outgroup = "fish") key$leads[, c("Statement", "Character", "Next", "Taxon")] classify(key, alignmentScores(key, aln)) } # the example files that ship with dimoose fasta <- system.file("extdata", "aligned.fasta", package = "dimoose") mtdna <- system.file("extdata", "mtdna.fasta", package = "dimoose") if (requireNamespace("ape", quietly = TRUE)) { keyFromAlignment(fasta) keyFromAlignment(fasta, tree = system.file("extdata", "aligned.nwk", package = "dimoose")) keyFromAlignment(mtdna, reference = "H2a2a1", outgroup = "L0a1") } ## Not run: keyFromAlignment(mtdna, tool = "iqtree", outgroup = "L0a1") # needs IQ-TREE installed ## End(Not run)aln <- c(human = "ACGTACGTACGA", chimp = "ACGTACGTATGA", mouse = "ACCTACATGCGA", rat = "ACCTACATGCGT", fish = "TCCTTCGTACCA") if (requireNamespace("ape", quietly = TRUE)) { key <- keyFromAlignment(aln, outgroup = "fish") key$leads[, c("Statement", "Character", "Next", "Taxon")] classify(key, alignmentScores(key, aln)) } # the example files that ship with dimoose fasta <- system.file("extdata", "aligned.fasta", package = "dimoose") mtdna <- system.file("extdata", "mtdna.fasta", package = "dimoose") if (requireNamespace("ape", quietly = TRUE)) { keyFromAlignment(fasta) keyFromAlignment(fasta, tree = system.file("extdata", "aligned.nwk", package = "dimoose")) keyFromAlignment(mtdna, reference = "H2a2a1", outgroup = "L0a1") } ## Not run: keyFromAlignment(mtdna, tool = "iqtree", outgroup = "L0a1") # needs IQ-TREE installed ## End(Not run)
Classes are clustered on the mean embedding of their images (Ward's method), giving a binary tree. Each split is then described by the vocabulary pair that best separates its two sides: the pair maximising |(L.yes - L.no) - (R.yes - R.no)| over the side means, not used above that split. The "Has feature" lead goes to the side the feature scores higher on. Machine answers use the same two prompts.
keyFromClusters( image, images, vocabulary, textEmb, template = "{x}", order = NULL, maxPerSide = 10, calibrate = c("none", "midpoint"), desc = "Key generated from image clusters", meta = NULL )keyFromClusters( image, images, vocabulary, textEmb, template = "{x}", order = NULL, maxPerSide = 10, calibrate = c("none", "midpoint"), desc = "Key generated from image clusters", meta = NULL )
image |
|
images |
|
vocabulary |
As in |
textEmb |
Text embeddings of |
template |
Must match the one used to make |
order |
Optional function of a character vector of labels returning a number; at each split the child with the smaller value comes first. |
maxPerSide |
Most images per side used for the side means. |
calibrate |
|
desc, meta
|
Title and metadata. |
A dimoose key with clip_text_pair features.
The common constructor behind sharkKey(), arachnidaKey(),
vibrioKey() and phylotreeKey(): takes one row per lead and returns a
dimoose object.
keyFromLeads(leads, desc = NA, meta = NULL, taxa = NULL, features = NULL)keyFromLeads(leads, desc = NA, meta = NULL, taxa = NULL, features = NULL)
leads |
Data frame with columns |
desc |
Title or description of the key. |
meta |
Key/value data frame of metadata (e.g. |
taxa |
Data frame of results with |
features |
Optional |
A dimoose object.
leads <- data.frame( Statement = c("1", "1", "2", "2"), Choice = c("a", "b", "a", "b"), Character = c("Wings present", "Wings absent", "Two wings", "Four wings"), Next = c("2", "-", "-", "-"), Taxon = c("", "Apterygota", "Diptera", "Other") ) k <- keyFromLeads(leads, "A tiny key") k k$toNewick()leads <- data.frame( Statement = c("1", "1", "2", "2"), Choice = c("a", "b", "a", "b"), Character = c("Wings present", "Wings absent", "Two wings", "Four wings"), Next = c("2", "-", "-", "-"), Taxon = c("", "Apterygota", "Diptera", "Other") ) k <- keyFromLeads(leads, "A tiny key") k k$toNewick()
Generates a dichotomous key from a taxon-by-character matrix, the form most identification data is kept in (DELTA, Lucid, SDD, a spreadsheet of test results). Unlike a classification tree, the key is built the way a person writes one: a taxon that is variable for a character is keyed out down both leads of every couplet that uses the character.
keyFromMatrix( matrix, taxon = "Taxon", characters = NULL, season = NULL, variable = c("both", "missing"), desc = "Key generated from a character matrix", meta = NULL, maxCouplets = 10000 )keyFromMatrix( matrix, taxon = "Taxon", characters = NULL, season = NULL, variable = c("both", "missing"), desc = "Key generated from a character matrix", meta = NULL, maxCouplets = 10000 )
matrix |
A character matrix: a data frame with one row per taxon (see
'The matrix'), e.g. the |
taxon |
Name of the column holding the taxon names. |
characters |
Optional data frame of character metadata with a column
|
season |
Optional character vector. Only characters whose |
variable |
|
desc |
Title of the key. |
meta |
Key/value metadata data frame. |
maxCouplets |
For |
A dimoose object. With variable = "both" its leads also carry
Question (the character's name), MatrixCharacter and MatrixStates
(the character and the /-separated states the lead accepts) and, if
characters has it, Needs.
A plain data frame (or a character matrix with column names) with one row
per taxon: a taxon column and one column per character. Cells are state
strings (anything else is converted with as.character()). A taxon that
is variable (polymorphic) for a character lists its states separated by
/, e.g. "+/-" or "red/white". NA or an empty cell means unknown or
inapplicable. If a taxon has several rows, they are merged and the taxon
takes every state seen in any of them. Characters with fewer than two
states among the taxa are dropped. Every column other than taxon is taken
as a character, so leave out columns of notes.
A key can only ask about states a specimen either shows or does not, so a
measurement is used through the ranges that do not overlap. In a column
whose cells are all numbers or ranges, as the readers readDELTA(),
readSDD(), readLucid() and readXper() write them ("3", "4-6",
"(1-)2-8(-12)", taken from the lowest to the highest number in the
cell), ranges that overlap or touch are merged into one state named by its
extremes: "3-5", "5-7" and "12-20" give the two states "3-7" and
"12-20". A character whose ranges all overlap has one state and is
dropped. A message says what was merged or dropped. The cells may end in a
unit if it is the same in every cell of the column ("2-8 mm",
"5-12 mm"). Negative numbers are
not understood (a - is read as "to"). To key a taxon out on part of an
overlapping range, cut the measurement into classes yourself and enter the
classes as states ("3/5" and "5/7" leaflets in the example below).
If characters has a column Type, as the tables of those readers do,
characters of type "text" are not used, the rule above applies to
"integer" and "real" characters only, and "categorical" and
"ordered" characters are always taken as states.
variable = "both")Every couplet tests one state of one character against the rest: the first
lead reads "Character: state", the second "Character: other or another", naming the other states the taxa still in play may show. A
character with three or more states is therefore keyed out over several
couplets, and every couplet has exactly two leads.
A taxon follows the first lead if it can show the state and the second if
it can show any other state not already ruled out, so a variable taxon
follows both. So does a taxon whose state is unknown (NA), since it
cannot be ruled out on either side. The guarantee is: whichever of its
states a specimen shows for each character, following the key ends at a
result that names its taxon.
At each step the couplet is chosen like this. For every state of every
character, count the taxa that follow the first lead only (a) and the
second lead only (b). The couplet separates a * b pairs of taxa; taxa
that follow both leads separate nothing. The split with the largest
a * b / Cost is taken; ties go to the character that comes first in the
matrix, then to the state that appears first in its column. With no
Cost, every character costs 1. A character with Cost 2 is thus chosen
over one with Cost 1 only if it separates more than twice as many pairs.
If no split separates a pair, the taxa that are left overlap: one shows
"+" and another "+/-", or one has 3 to 5 leaflets and another 5 to 7.
A couplet can still rule taxa out on one side ("-" can only be the
second taxon), so the split with the largest (a + b) / Cost is taken,
provided some taxon is known to show the state and some taxon is known to
show another. When no such split is left either, the remaining taxa share
a result, written "A / B": with the states observed they cannot be told
apart.
A group of taxa that is reached by several routes with the same states still open is keyed out once, and couplets that come out the same lead for lead are merged, so a couplet may be the target of more than one lead, and a variable taxon appears at more than one result.
If characters has a column InapplicableIf ("Other character=state"
or "Other character=state/state", several rules separated by ;, as
written by readSDD() and readDELTA()), a character is not asked about
while a taxon still in play may show one of those states of the other
character: the key asks whether there is an aerial stem before it asks
whether the stem is hairy. The rule is dropped where it would leave no
couplet at all.
Because variable and unknown taxa follow both leads, the key grows with
the number of such cells. Random two-state matrices of 200 taxa and 60
characters gave 199 couplets with no unknown cells, about 550 with 30%
unknown and about 3400 with 50%. A matrix in which most characters are
known for only a few taxa can need a number of couplets that doubles with
every character; maxCouplets stops such a key with an error rather than
running out of memory, and a key with more than ten couplets per taxon
comes with a warning. Leave out the characters that are unknown for most
taxa, or use variable = "missing".
variable = "missing"Variable cells are treated as unknown and the key is the classification
tree fitted by rpart::rpart() (through keyFromRpart()), with Cost
passed as rpart's cost. Each taxon then appears at one result, but rpart
places a taxon it has no state for with surrogate splits, so a specimen
showing the taxon's other state is keyed out wrongly. A taxon rpart cannot
place at a split at all is listed at every result below that split (a
message says how many such taxa there are). The
key is smaller; use it when that matters more than the guarantee above. A
lead may name several states ("Colour: red,white"). InapplicableIf is
not used.
vibrioKey(), vibrioMatrix(), keyFromRpart(), generateTree()
# Some of Montana's broadleaf trees (see montanaKey() for the key written # by hand). Box elder has 3 to 5 leaflets and green ash 5 to 7, so a leaf # with 5 could be either. trees <- data.frame( Taxon = c("Box elder", "Green ash", "Rocky Mountain maple", "Bur oak", "Quaking aspen", "Paper birch", "Water birch"), `Buds and leaves` = c("opposite", "opposite", "opposite", "alternate", "alternate", "alternate", "alternate"), `Leaf type` = c("compound", "compound", "simple", "simple", "simple", "simple", "simple"), Leaflets = c("3/5", "5/7", NA, NA, NA, NA, NA), `Leaf lobes` = c(NA, NA, "lobed", "lobed", "not lobed", "not lobed", "not lobed"), Bark = c(NA, NA, NA, NA, "smooth and pale", "white and peeling", "gray to brown"), check.names = FALSE ) k <- keyFromMatrix(trees, desc = "Some Montana trees") k$leads[c("Statement", "Choice", "Character", "Next", "Taxon")] # Leaves are easy to see but only there in summer; bud arrangement needs a # closer look but works all year characters <- data.frame( Character = c("Buds and leaves", "Leaf type", "Leaflets", "Leaf lobes", "Bark"), Cost = c(2, 1, 1, 1, 1), Season = c("", "summer", "summer", "summer", ""), Needs = c("a twig in the hand", "", "", "", "") ) keyFromMatrix(trees, characters = characters)$leads$Character # A winter key: what bark and buds cannot separate shares a result winter <- keyFromMatrix(trees, characters = characters, season = "winter") winter$leads[c("Character", "Taxon")] # A matrix read from a DELTA data set, with its character table. Body # length is used because the three ranges do not overlap; the comments are # text and are left out. beetles <- readDELTA(system.file("extdata", "tdwg", "beetles-delta", package = "dimoose")) keyFromMatrix(beetles$matrix, characters = beetles$characters)$leads[c("Character", "Taxon")]# Some of Montana's broadleaf trees (see montanaKey() for the key written # by hand). Box elder has 3 to 5 leaflets and green ash 5 to 7, so a leaf # with 5 could be either. trees <- data.frame( Taxon = c("Box elder", "Green ash", "Rocky Mountain maple", "Bur oak", "Quaking aspen", "Paper birch", "Water birch"), `Buds and leaves` = c("opposite", "opposite", "opposite", "alternate", "alternate", "alternate", "alternate"), `Leaf type` = c("compound", "compound", "simple", "simple", "simple", "simple", "simple"), Leaflets = c("3/5", "5/7", NA, NA, NA, NA, NA), `Leaf lobes` = c(NA, NA, "lobed", "lobed", "not lobed", "not lobed", "not lobed"), Bark = c(NA, NA, NA, NA, "smooth and pale", "white and peeling", "gray to brown"), check.names = FALSE ) k <- keyFromMatrix(trees, desc = "Some Montana trees") k$leads[c("Statement", "Choice", "Character", "Next", "Taxon")] # Leaves are easy to see but only there in summer; bud arrangement needs a # closer look but works all year characters <- data.frame( Character = c("Buds and leaves", "Leaf type", "Leaflets", "Leaf lobes", "Bark"), Cost = c(2, 1, 1, 1, 1), Season = c("", "summer", "summer", "summer", ""), Needs = c("a twig in the hand", "", "", "", "") ) keyFromMatrix(trees, characters = characters)$leads$Character # A winter key: what bark and buds cannot separate shares a result winter <- keyFromMatrix(trees, characters = characters, season = "winter") winter$leads[c("Character", "Taxon")] # A matrix read from a DELTA data set, with its character table. Body # length is used because the three ranges do not overlap; the comments are # text and are left out. beetles <- readDELTA(system.file("extdata", "tdwg", "beetles-delta", package = "dimoose")) keyFromMatrix(beetles$matrix, characters = beetles$characters)$leads[c("Character", "Taxon")]
Converts an rpart::rpart() classification tree into a dichotomous key:
every split becomes a couplet with two leads, and every leaf a result.
Taxa that the tree cannot separate share a leaf and are listed together
("A / B"). A taxon rpart could not send down either side of a split,
because the split's variable is missing for it, is listed at every result
below that split.
keyFromRpart( fit, desc = "Key generated from a classification tree", meta = NULL )keyFromRpart( fit, desc = "Key generated from a classification tree", meta = NULL )
fit |
An |
desc |
Title of the key. |
meta |
Key/value metadata data frame. |
A dimoose object.
keyFromMatrix(), which builds a key straight from a character
matrix and handles variable taxa; vibrioKey()
d <- data.frame( taxon = factor(c("A", "B", "C")), wings = factor(c("+", "-", "-")), spots = factor(c("-", "+", "-")) ) fit <- generateTree(d, "taxon") keyFromRpart(fit, "Three taxa")$leadsd <- data.frame( taxon = factor(c("A", "B", "C")), wings = factor(c("+", "-", "-")), spots = factor(c("-", "+", "-")) ) fit <- generateTree(d, "taxon") keyFromRpart(fit, "Three taxa")$leads
Each step asks whether the image has one sniglet. method = "balanced" is fordera's algorithm: a sniglet is present when its score is
above the quantile of that sniglet's scores over all images; a label has
a sniglet if any of its images does; each split uses the unused sniglet
that divides the remaining labels most evenly (ties to the earlier one).
method = "rpart" suits several images per label. It first gives each
sniglet one threshold, the cut of its scores that best separates the
labels, and then grows a classification tree on has/lacks, so a sniglet
means the same thing wherever the key asks about it.
keyFromSniglets( scores, images, sniglets, method = c("balanced", "rpart"), quantile = 0.6, desc = "Key generated from sniglets", meta = NULL ) keyFromTerms(scores, images, terms, ...)keyFromSniglets( scores, images, sniglets, method = c("balanced", "rpart"), quantile = 0.6, desc = "Key generated from sniglets", meta = NULL ) keyFromTerms(scores, images, terms, ...)
scores |
|
images |
|
sniglets |
The |
method |
|
quantile |
Presence threshold quantile for |
desc, meta
|
Title and metadata of the key. |
terms, ...
|
|
A dimoose key whose leads carry Feature, Test, Threshold and
Question, and whose features table holds the centroids. Feature ids
are "sniglet:<coined word>" and never change; label is the name
shown, which renameSniglets() changes.
A lead that only says a specimen lacks something gives the reader nothing
to look for. Where it can, the "Lacks" lead therefore also names what its
images have instead: another sniglet that at least 90% of them have and at
most 10% of the other side do ("Lacks gaithiark; has blienvaurk"). The
machine still tests only the first sniglet. Every lead also records example
regions from the images that take it (Examples), which
patchExemplars() turns into crops: for a "Lacks" lead with no such second
sniglet, they are the regions that come closest to the missing one.
Reads a key written in a spreadsheet and saved as CSV (in Excel or Google Sheets: File > Save as / Download > CSV). Each row is one lead:
keyFromSpreadsheet( file, photos = NULL, desc = NA, meta = NULL, checkNames = FALSE )keyFromSpreadsheet( file, photos = NULL, desc = NA, meta = NULL, checkNames = FALSE )
file |
Path of the CSV file (UTF-8). |
photos |
Folder holding the photos named in the |
desc |
Title of the key. |
meta |
Optional key/value metadata data frame (e.g. a |
checkNames |
If |
Couplet: the couplet's number.
Description: what the lead says ("Leaves compound.").
Goes to: the couplet this lead leads to, or blank when the lead
ends in a species.
Species: the result, for leads that end the key. Write it as
"Common name (Genus species)" to have keyDiagram() set the
scientific name in italics.
Choice (optional): the lead's letter; by default the leads of each
couplet are lettered a, b, c in the order they appear (after z come
aa, ab and so on).
Question (optional): what the couplet asks ("Leaf type?"), shown by
keyDiagram() and exportWizard().
Photo (optional): file names of photos to show with the lead,
separated by commas or semicolons; the extension may be left out. A
web address (https://...) is kept as a link to the picture instead.
Note and Label (optional): a remark on the lead, as pruneKey()
writes, and a label to show for the lead in place of couplet and
letter. Both are kept in the key's leads as they are.
Column names are matched loosely (case, spaces and punctuation do not
matter) and dimoose's own names work too: Statement, Character,
Next, Taxon and Image. Other columns are ignored and blank rows are
skipped. The photos are embedded in the key, so the wizard and the
diagram work offline.
The file may be as Excel saves it: with or without a byte-order mark, in UTF-8 or Windows-1252, and with semicolons between the cells, which is what Excel writes where the comma is the decimal mark.
The Montana key that comes with dimoose is a ready-made example to copy:
system.file("extdata", "montana", "montana-key.csv", package = "dimoose"),
with its photos in the same folder.
A dimoose key.
keyToSpreadsheet() to write a key as such a spreadsheet,
keyFromLeads() to build a key from a data frame,
keyDiagram() and exportWizard() to show it.
csv <- system.file("extdata", "montana", "montana-key.csv", package = "dimoose") k <- keyFromSpreadsheet(csv, desc = "Native broadleaf trees of Montana") k keyDiagram(k, tempfile(fileext = ".svg"))csv <- system.file("extdata", "montana", "montana-key.csv", package = "dimoose") k <- keyFromSpreadsheet(csv, desc = "Native broadleaf trees of Montana") k keyDiagram(k, tempfile(fileext = ".svg"))
Most identification keys exist only as printed text: floras, monographs,
scans in the Biodiversity Heritage Library. keyFromText() reads such a
key, one lead at a time (couplet, lead, text and destination), builds a
dimoose key with keyFromLeads() and checks it with $validate().
parseTextKey() is the lower-level half that returns the lead table and
the list of problems without building the key.
keyFromText(text, desc = NA, meta = NULL, llm = NULL, strict = FALSE, ...)keyFromText(text, desc = NA, meta = NULL, llm = NULL, strict = FALSE, ...)
text |
The key: a file path, one string holding the whole key, or a character vector of lines. Give the key alone, without the descriptions before and after it. |
desc |
Title of the key. By default the first line above the first lead is used, if there is one. |
meta |
Key/value data frame of metadata (e.g. |
llm |
Optional function for a first pass over messy text; see the section below. |
strict |
If |
... |
Further arguments for |
Layouts that are recognised:
Bracketed keys, where the leads of a couplet stand together:
1a./1b., 1./1'. (primes), 1./1., 1/- (a dash for the
second lead), 1(a)/1(b), (1)/(1), Roman numerals II.,
A./AA. or a./aa. lettering, and labels with a back-reference
such as 3(1). (couplet 3, reached from couplet 1). The second lead
may also be marked +, or carry no label at all: a new paragraph, as
in many older entomological keys, or a line of its own with dot
leaders under a complete lead.
Indented (yoked) keys, where the second lead of a couplet comes much later and a lead without a destination leads to the couplet printed under it.
Indented keys without any couplet numbers, as in most older floras:
the leads of a couplet share an indent, and a lead ends in a name,
often with its number in the flora (... long-acuminate. 1. S. nigra.).
The couplets are made from the indentation and numbered in order.
Plain OCR text usually loses the indentation. The couplets are then
worked out from the order of the leads, which works when every couplet
has exactly two leads: a lead without a name opens the next couplet,
a lead with a name is followed by the second lead of the nearest
couplet still lacking one. This is reported as a repair; if the leads
do not pair off, the key is refused with the line where the pairing
breaks. One key at a time: a second heading in the text stops the
reading there.
Paired keys, where every lead has its own number and names its
alternative in parentheses (1 (8) ... and further down 8 (1) ...).
The two are put together as one couplet, named after the lower number.
Couplets with three or more leads, and couplets reached from several leads.
The destination is whatever follows dot leaders, a tab, a run of three
or more spaces, a colon before a number (lobed: 5) or a spaced dash
(a large tree -- Quercus alba): a couplet number (also go to 5,
see 5, (5), > 5) or a taxon. A taxon may carry an authority, a
common name (in parentheses, or in front of the scientific name) and a
page number (Salix, p. 212, HEMIPTERA (Page 76), Culicidae, 96);
the genus may be abbreviated (S. alba L.). These go to the
Authority, CommonName and Page columns of the lead table. A number
in parentheses after a name (Pinus. (7).) is taken for its page or
list number, not for a couplet. Lead text may wrap over several lines,
with the destination on the last one; words hyphenated at a line end are
joined, and blank lines, page numbers, running heads and figure captions
in between are skipped, as are footnotes. Text in two columns side by
side (as pdftotext -layout gives it) is read column by column; this
is reported as a repair. Markdown emphasis and tables are read as plain
text.
Scanned text is repaired where that is safe: l, I or | read as 1
and O as 0 in couplet numbers, labels that lost their full stop,
labels garbled beyond reading where only one couplet can be meant
(V0. between 9 and 11), and destinations that lost their dot leaders
(... each year. 11., ... leaves flat. Abies.). Text with two or more
spaces between all words, as many scans have, is recognised, and only
clearly wider gaps count as gaps there. Every repair is reported, and
with strict = TRUE the function stops instead of repairing.
Nothing is dropped silently. A lead without a destination, a couplet
with one lead, a destination couplet that does not exist, a destination
that is probably a broken number (S for 5), a jump in the couplet
numbers, a label that stands alone on its line, text that looks like a
lead without a label and similar faults are reported with the line they
are on. keyFromText() gives one warning that lists them and attaches
the full table to the key as attr(key, "problems"). Text that was
passed over (running heads, descriptions between leads) is listed there
as notes.
What a scan has lost cannot be put back: a couplet number misread as
another number that also exists in the key gives a wrong link without a
warning, and leads whose labels the scan put in a column of their own
are reported as lost, not recovered. Check a key read from a scan
against the page. Not handled: keys without numbers that are not
indented and have couplets of three or more leads, keys that mix
letters and unnumbered leads (Gray's Manual), keys divided by section
headings that stand for leads (A., *, ** in Britton & Brown), and
text in more than two columns.
A dimoose key. attr(key, "problems") is a data frame of
problems, repairs and notes (see parseTextKey()).
For badly scanned keys, llm can be a function that takes the whole text
as one string and returns either cleaned-up key text (one lead per line,
in any layout above), a data frame with columns couplet, lead,
text and destination, or the same as JSON (an array of leads; needs
the jsonlite package). Markdown code fences around the answer are taken
off. You write that function; dimoose calls no model or API itself. What
comes back is not trusted: it goes through the same checks, and each lead
is compared with the source, so a lead whose words or taxon do not occur
in the source, a destination that is not on the lines of its lead in the
source (two names swapped, a lead sent to another couplet), or a lead the
model left out, is reported with the line of the source it belongs to
(a lead that is not in the source at all has no such line; the message
then names the line or row of the model's answer). The title is taken
from the source, not from the answer.
An answer that cannot be read (cut-off JSON, prose without leads, an
empty table) is an error; nothing is taken from it.
parseTextKey(), keyFromLeads()
# A bracketed key with dot leaders, wrapped leads and authorities file <- system.file("extdata", "textkeys", "trees-bracketed.txt", package = "dimoose") cat(readLines(file, n = 8), sep = "\n") trees <- keyFromText(file) trees trees$leads[, c("Statement", "Choice", "Next", "Taxon", "CommonName")] # An indented key, given as a string conifers <- keyFromText(system.file("extdata", "textkeys", "conifers-indented.txt", package = "dimoose")) conifers$toNewick() # Problems are reported with their line and attached to the key broken <- "1a. Wings present .... 2 1b. Wings absent .... Apterygota 2a. Two wings .... Diptera 2b. Four wings" k <- suppressWarnings(keyFromText(broken, "Insects")) attr(k, "problems")[, c("Line", "Lead", "Kind", "Message")] ## Not run: # A first pass by a language model. Any function(text) will do; this one # calls a chat API and asks for one lead per line. askModel <- function(text) { prompt <- paste( "This is a scanned identification key. Rewrite it with one lead per", "line as '<couplet><letter>. <text> .... <destination>'. Do not add,", "drop or reword leads.\n\n", text ) response <- httr::POST( "https://api.example.com/v1/messages", httr::add_headers(Authorization = paste("Bearer", Sys.getenv("MODEL_API_KEY"))), body = list(model = "some-model", messages = list(list(role = "user", content = prompt))), encode = "json" ) httr::content(response)$content[[1]]$text } key <- keyFromText("scanned-key.txt", llm = askModel) attr(key, "problems") ## End(Not run)# A bracketed key with dot leaders, wrapped leads and authorities file <- system.file("extdata", "textkeys", "trees-bracketed.txt", package = "dimoose") cat(readLines(file, n = 8), sep = "\n") trees <- keyFromText(file) trees trees$leads[, c("Statement", "Choice", "Next", "Taxon", "CommonName")] # An indented key, given as a string conifers <- keyFromText(system.file("extdata", "textkeys", "conifers-indented.txt", package = "dimoose")) conifers$toNewick() # Problems are reported with their line and attached to the key broken <- "1a. Wings present .... 2 1b. Wings absent .... Apterygota 2a. Two wings .... Diptera 2b. Four wings" k <- suppressWarnings(keyFromText(broken, "Insects")) attr(k, "problems")[, c("Line", "Lead", "Kind", "Message")] ## Not run: # A first pass by a language model. Any function(text) will do; this one # calls a chat API and asks for one lead per line. askModel <- function(text) { prompt <- paste( "This is a scanned identification key. Rewrite it with one lead per", "line as '<couplet><letter>. <text> .... <destination>'. Do not add,", "drop or reword leads.\n\n", text ) response <- httr::POST( "https://api.example.com/v1/messages", httr::add_headers(Authorization = paste("Bearer", Sys.getenv("MODEL_API_KEY"))), body = list(model = "some-model", messages = list(list(role = "user", content = prompt))), encode = "json" ) httr::content(response)$content[[1]]$text } key <- keyFromText("scanned-key.txt", llm = askModel) attr(key, "problems") ## End(Not run)
Counts the steps (couplets answered) it takes to reach each result of a key and compares the average with the fewest steps any key could need.
keyQuality(key, weights = NULL)keyQuality(key, weights = NULL)
key |
A dimoose key. |
weights |
How common each taxon is: a numeric vector whose names
are the key's results as written in its |
A key is a code: each result is spelled by the leads chosen on the way to
it. bound is the expected number of steps of the best key there could
be for the same taxa and the same weights, among keys whose couplets
have no more leads than the largest couplet of this key (maxLeads). It
is found by Huffman's construction, which joins the rarest taxa first, so
it is a number of steps that a key can really reach: 3 for eight equally
common taxa and couplets of two leads, about 3.55 for eleven. No key can
do better, however its characters are chosen. efficiency is
bound / expectedSteps: 1 for a key that could not be shorter, and lower
the more steps the key takes than it would have to. Real keys rarely
reach 1, because the characters that would split the taxa evenly may not
exist.
When some taxa are met more often than others (weights), a good key
asks few questions before reaching the common ones, and the bound takes
the weights into account in the same way.
In a key that has a few large couplets among many small ones (a
PhyloTree key has couplets of 2 and of 87 leads), the best key is allowed
the largest couplet everywhere, so the bound is far below what the key
does and efficiency is low. Three more numbers describe such keys
without that comparison. entropy is the Shannon entropy of the weights
in bits (log2 of the number of taxa when they are equally common): the
information a key has to gather. A couplet with k leads can give up to
log2(k) bits, and bitsPerStep is the average of that over the steps
this key takes (1 for a key of two-lead couplets). capacityUsed is
entropy / (bitsPerStep * expectedSteps): the share of the information
its couplets could give that the key needs. It is 1 when every couplet
splits what reaches it evenly, whatever its size, and it is not a
comparison with a shorter key.
Two cases need a rule:
A taxon that keys out at several terminal leads counts once: its steps are the mean over its leads, as if its specimens were split evenly among them, and its weight is not multiplied by the number of leads.
A couplet reached from several leads (a reticulate key) is counted by
its shortest route from the first couplet, so steps is the fewest
steps in which a result can be reached. depth is the longest route.
Leads in couplets that cannot be reached from the first couplet, and terminal leads that name no taxon, are left out of the averages.
A list with
taxa: a data frame with one row per taxon: Taxon, leads (how
many terminal leads end in it), steps (mean over those leads),
minSteps, maxSteps and weight (its share of the weights, summing
to 1);
leads: a data frame with one row per terminal lead: Lead (its
label, e.g. "7b"), Statement, Choice, Taxon and steps;
meanLeadSteps: mean steps over terminal leads;
meanTaxonSteps: mean steps over taxa;
expectedSteps: mean steps over taxa weighted by weight (the same
as meanTaxonSteps without weights);
bound: the expected steps of the best possible key to these taxa
with these weights and couplets of at most maxLeads leads;
efficiency: bound / expectedSteps, above 0 and at most 1 (NA
for a key to a single taxon, which needs no steps);
entropy: the Shannon entropy of weight, in bits;
bitsPerStep: the most a step of this key can tell, on average
(1 when every couplet has two leads);
capacityUsed: entropy / (bitsPerStep * expectedSteps), at most 1;
maxLeads: the number of leads in the key's largest couplet;
depth: the longest route to a result, in steps.
summary() on a key, which reports the aggregates;
occurrenceWeights() for weights from GBIF.
q <- keyQuality(arachnidaKey()) q$taxa q$expectedSteps q$bound q$efficiency # spiders and mites are met far more often than the other orders common <- c(Araneae = 50, Acari = 40, Opiliones = 5, Scorpiones = 3) suppressWarnings(keyQuality(arachnidaKey(), common))$expectedStepsq <- keyQuality(arachnidaKey()) q$taxa q$expectedSteps q$bound q$efficiency # spiders and mites are met far more often than the other orders common <- c(Araneae = 50, Acari = 40, Opiliones = 5, Scorpiones = 3) suppressWarnings(keyQuality(arachnidaKey(), common))$expectedSteps
Writes a key as the CSV file that keyFromSpreadsheet() reads, so that a
key imported from FishBase or generated from data can be opened in Excel
or Google Sheets, edited by hand and read back. The columns are those of
the Montana template: Couplet, Choice, Question, Description,
Goes to, Species and Photo, followed by Note and Label when
the key's leads have them (a pruned key has notes, see pruneKey(); a
PhyloTree key has labels).
keyToSpreadsheet(key, file, photos = NULL, bom = TRUE)keyToSpreadsheet(key, file, photos = NULL, bom = TRUE)
key |
A dimoose key. |
file |
Path of the CSV file to write (UTF-8). |
photos |
Folder to write the key's embedded photos into; by default the folder the CSV file is in. It is created if needed. |
bom |
Start the file with a byte-order mark? Excel needs one to
show accented and other non-English letters correctly when the file is
opened by double-clicking; Google Sheets and |
Photos embedded in the key (see imageMeta()) are written as image files
into photos, named by their ids, and the Photo column names those
files. A file of that name that is already there with different contents
is left alone and used as it is, so writing a key back next to its
original photos does not replace them with the smaller embedded copies.
Pictures a lead refers to by web address stay web addresses.
Reading the file back gives the same leads and the same photo ids. The
exceptions are small: a photo id that cannot be a file name (it has a
/, :, comma or the like in it, or differs from another only by
capitals) is changed, with a message; a line break inside a cell comes
back as a plain line feed whatever it was; and leads without a Choice
are lettered.
A spreadsheet holds only those columns. Anything else in the key's
leads (for example the taxon links of a FishBase key, or the tests of a
key a machine can follow) is not written, and a message says which
columns were left out; the title and citation are not written either, so
pass desc and meta again to keyFromSpreadsheet().
file, invisibly, so the result can go straight to
keyFromSpreadsheet().
dir <- tempfile("key") dir.create(dir) csv <- keyToSpreadsheet(montanaKey(), file.path(dir, "montana.csv")) list.files(dir) # ... edit the file in a spreadsheet program, then read it back again <- keyFromSpreadsheet(csv, desc = "Montana trees, edited") identical(again$leads$Character, montanaKey()$leads$Character)dir <- tempfile("key") dir.create(dir) csv <- keyToSpreadsheet(montanaKey(), file.path(dir, "montana.csv")) list.files(dir) # ... edit the file in a spreadsheet program, then read it back again <- keyFromSpreadsheet(csv, desc = "Montana trees, edited") identical(again$leads$Character, montanaKey()$leads$Character)
Leave-one-out evaluation of a key builder
looKey(images, build, score, groups = NULL)looKey(images, build, score, groups = NULL)
images |
|
build |
Function of the training row indices returning a key. |
score |
Function of (key, test row index) returning a one-row scores matrix for that image. |
groups |
As in |
As evaluateKey().
Macrohaplogroup of a haplogroup
macroHaplogroup( haplogroup, level = c("letter", "depth"), depth = 2, tree = dimoose::phylotree17 )macroHaplogroup( haplogroup, level = c("letter", "depth"), depth = 2, tree = dimoose::phylotree17 )
haplogroup |
Haplogroup names. |
level |
|
depth |
For |
tree |
Default phylotree17. |
A character vector the length of haplogroup.
macroHaplogroup(c("H2a2a1", "L3e1", "U5b")) macroHaplogroup("H2a", level = "depth", depth = 3)macroHaplogroup(c("H2a2a1", "L3e1", "U5b")) macroHaplogroup("H2a", level = "depth", depth = 3)
Joins two keys end to end: the lead of key that ended in the taxon at
now leads to the first couplet of subkey. Use it to put a key to the
genera of one family under a key to families, or a regional key under a
general one.
mergeKeys(key, subkey, at, labels = c("renumber", "prefix"), prefix = NULL)mergeKeys(key, subkey, at, labels = c("renumber", "prefix"), prefix = NULL)
key |
A dimoose key. |
subkey |
The dimoose key to graft onto it. |
at |
The result of |
labels |
How to keep couplet names unique. |
prefix |
Text put in front of the subkey's couplet names with
|
The leads of the two keys are put in one table (a column that only one of
them has is left blank for the other), and so are their metadata, embedded
images, taxa and features. key's title is kept; subkey's citation is
added to key's. If at is the result of several leads, they all lead to
the subkey. An embedded image or a feature that the two keys both have
under one id, with different contents, is kept apart by giving the
subkey's a new id (_2 is added), with a warning for a feature.
A dimoose key.
families <- keyFromLeads(data.frame( Statement = c("1", "1"), Choice = c("a", "b"), Character = c("Fruit an acorn", "Fruit a capsule releasing cottony seeds"), Next = "-", Taxon = c("Fagaceae", "Salicaceae") ), "Two families of trees") salicaceae <- keyFromLeads(data.frame( Statement = c("1", "1"), Choice = c("a", "b"), Character = c("Buds with one scale", "Buds with several scales"), Next = "-", Taxon = c("Salix", "Populus") ), "Genera of Salicaceae") both <- mergeKeys(families, salicaceae, at = "Salicaceae") both$leads mergeKeys(families, salicaceae, at = "Salicaceae", labels = "prefix")$leads$Statementfamilies <- keyFromLeads(data.frame( Statement = c("1", "1"), Choice = c("a", "b"), Character = c("Fruit an acorn", "Fruit a capsule releasing cottony seeds"), Next = "-", Taxon = c("Fagaceae", "Salicaceae") ), "Two families of trees") salicaceae <- keyFromLeads(data.frame( Statement = c("1", "1"), Choice = c("a", "b"), Character = c("Buds with one scale", "Buds with several scales"), Next = "-", Taxon = c("Salix", "Populus") ), "Genera of Salicaceae") both <- mergeKeys(families, salicaceae, at = "Salicaceae") both$leads mergeKeys(families, salicaceae, at = "Salicaceae", labels = "prefix")$leads$Statement
Model details as metadata rows
modelMeta(model)modelMeta(model)
model |
A key/value data frame with model_* keys.
A dichotomous key to the eleven native broadleaf (hardwood) trees of
Montana, as a table of leads (see keyFromLeads()); montanaKey() turns it
into a dimoose key with photos. Written for dimoose: native status was
checked in the Montana Field Guide and USDA PLANTS, and every couplet's
characters come from Montana Field Guide or Virginia Tech Dendrology
descriptions. Common hackberry is left out because the Montana Field Guide
gives its origin in Montana as undetermined.
montanamontana
A data frame with 20 rows (10 couplets) and 7 character columns:
Couplet number.
a or b.
Text of the lead.
Couplet the lead leads to, or "-".
For terminal leads, "Common name (Genus species)".
What the couplet asks, e.g. "Leaf type?".
Ids of the photos in inst/extdata/montana/ shown with the
lead, separated by ;.
Montana Field Guide, Montana Natural Heritage Program and Montana
Fish, Wildlife & Parks, https://fieldguide.mt.gov; USDA PLANTS
Database, https://plants.usda.gov; Virginia Tech Dendrology fact sheets,
https://dendro.cnre.vt.edu. The page for each species and the source of
each couplet are listed in data-raw/montana/sources.md in the package
sources.
Counts the occurrence records GBIF holds for each
result of a key, optionally within a country or region. The counts can be
given to keyQuality() or summary() as weights, to see how many
steps the key takes for the taxa people actually meet.
occurrenceWeights( key, region = NULL, source = "gbif", fetch = qualGbifFetch, kingdom = NULL )occurrenceWeights( key, region = NULL, source = "gbif", fetch = qualGbifFetch, kingdom = NULL )
key |
A dimoose key. |
region |
Where to count: |
source |
Where the counts come from; only |
fetch |
The function that asks GBIF: called as |
kingdom |
Optional kingdom ("Plantae", "Animalia", ...), which helps GBIF tell apart names that are used in more than one kingdom. |
A result written as "Common name (Genus species)" is looked up by the
name in parentheses; other results by the result itself when it looks
like a scientific name, which includes genera and families ("Salix",
"Salix sp." and "Salix spp." all count the genus) and names followed
by their author ("Puma concolor (Linnaeus, 1771)"). A result that
lists several taxa the key does not separate, written "A / B" as
keyFromRpart() and keyFromMatrix() write them, gets the sum of their
counts. Each name is matched to the GBIF Backbone Taxonomy, and the
records of the name GBIF accepts for it are counted, so a synonym gets
the count of its accepted name. This makes two requests to GBIF per
name, one name at a time; after three failed requests in a row the rest
are not tried.
A name that GBIF knows in more than one kingdom ("Morus" is a mulberry
and a gannet) is not counted unless kingdom says which is meant.
Occurrence counts measure how often a taxon is recorded, which follows how conspicuous and how popular with observers it is as well as how common it is. A taxon with no records gets a weight of 0 and so does not count towards the expected steps at all.
A numeric vector of record counts named by the key's results (as
in its Taxon column), NA for results that have no scientific name,
that GBIF does not know or cannot tell apart, or whose request failed
(each with a warning).
# canned answers in place of GBIF, to show the shape of `fetch` counts <- c("Acer negundo" = 61000, "Fraxinus pennsylvanica" = 48000) canned <- function(url, query) { if (grepl("species/match", url)) { list(usageKey = match(query$name, names(counts)), matchType = "EXACT") } else { list(count = unname(counts[query$taxonKey])) } } leads <- data.frame( Statement = "1", Choice = c("a", "b"), Character = c("Leaflets 3.", "Leaflets 5 to 7."), Next = "-", Taxon = c("Box elder (Acer negundo)", "Green ash (Fraxinus pennsylvanica)") ) occurrenceWeights(keyFromLeads(leads), fetch = canned) k <- montanaKey() w <- occurrenceWeights(k, region = "USA.27_1", kingdom = "Plantae") w summary(k, weights = w)# canned answers in place of GBIF, to show the shape of `fetch` counts <- c("Acer negundo" = 61000, "Fraxinus pennsylvanica" = 48000) canned <- function(url, query) { if (grepl("species/match", url)) { list(usageKey = match(query$name, names(counts)), matchType = "EXACT") } else { list(count = unname(counts[query$taxonKey])) } } leads <- data.frame( Statement = "1", Choice = c("a", "b"), Character = c("Leaflets 3.", "Leaflets 5 to 7."), Next = "-", Taxon = c("Box elder (Acer negundo)", "Green ash (Fraxinus pennsylvanica)") ) occurrenceWeights(keyFromLeads(leads), fetch = canned) k <- montanaKey() w <- occurrenceWeights(k, region = "USA.27_1", kingdom = "Plantae") w summary(k, weights = w)
Models are large (tens to hundreds of megabytes), so they are kept outside
your project, in R's per-user cache folder for dimoose. Set the option
dimoose.onnx.cache to use another folder.
onnxCache()onnxCache()
A path. The folder is not created until a model is stored in it.
onnxCache()onnxCache()
The light way to embed images and texts: the Python side needs only
onnxruntime, numpy, pillow and tokenizers (about 150 MB installed,
against 0.8 GB to several GB for PyTorch, depending on the platform). The
result is used exactly like a visionModel(): with embedImages(),
embedTexts(), scoreImages(), suggestSpecies() and the rest.
visionModel(backend = "onnx") calls this function.
onnxModel( name = "clip-vit-base-patch32", precision = c("fp32", "fp16", "int8"), textPrecision = precision, text = TRUE, cache = onnxCache(), python = NULL, threads = 0, fetch = NULL )onnxModel( name = "clip-vit-base-patch32", precision = c("fp32", "fp16", "int8"), textPrecision = precision, text = TRUE, cache = onnxCache(), python = NULL, threads = 0, fetch = NULL )
name |
The model: a folder, a Hugging Face repository or a known name (see below). |
precision |
|
textPrecision |
The same for the text encoder; by default as
|
text |
If |
cache |
Folder for downloaded models. |
python |
Optional path to the Python executable to use (passed to
|
threads |
Number of CPU threads for ONNX Runtime; |
fetch |
For tests: a function |
An object of class dimooseVisionModel, as visionModel() returns,
with a files entry: the model's folder (dir), its image and text
encoder files (image, text) and, for a model from the Hub, its
repository (repo) and the web address of the image encoder (imageUrl).
name is one of
a folder on your computer that holds an ONNX model (also written
"local-dir:<folder>"), in either layout described below;
a Hugging Face repository, "<organisation>/<model>" or
"hf-hub:<organisation>/<model>". The files that are needed are
downloaded once into onnxCache() and reused; nothing is downloaded when
they are already there;
"clip-vit-base-patch32" (the default; CLIP ViT-B/32, the same weights as
visionModel()) or "clip-vit-base-patch16": the ready-made exports
Xenova/clip-vit-base-patch32 and Xenova/clip-vit-base-patch16 on the
Hub;
"mobileclip-s0": Apple's MobileCLIP-S0 (Xenova/mobileclip_s0), a
small model for phones. Its image encoder is 46 MB at "fp32" and 23 MB
at "fp16". Its "int8" file is refused: its embeddings are wrong
(measured: cosine 0.09 on average with the "fp32" ones);
"bioclip", "bioclip-2" or any other model you have converted with
exportOnnxModel(). There is no official ONNX copy of BioCLIP on the
Hub, so it has to be converted once on a computer with PyTorch; the
converted folder can then be copied to computers without it.
Each encoder comes in up to three precisions. Measured on dimoose's 55 example photos against the PyTorch model (CLIP ViT-B/32 image encoder):
precision |
image encoder | cosine with PyTorch (lowest) |
"fp32" |
352 MB | 1.0000 |
"fp16" |
176 MB | 1.0000 |
"int8", converted by exportOnnxModel() |
96 MB | 0.998 |
"int8", Xenova/clip-vit-base-patch32 (_quantized) |
89 MB | 0.81 |
"fp16" halves the download but runs up to three times slower on a CPU
than "fp32". BioCLIP converted by exportOnnxModel() is 345, 173 and
89 MB, with cosines of 1.0000, 1.0000 and 0.998 (BioCLIP 2, a larger
model, is about 1217, 609 and 310 MB); on the Montana example photos all
three precisions name the same species. A key must be followed with the
model and precision it was built with when the precision is "int8":
its embeddings differ slightly from the others'. The ready-made "int8"
file of Xenova/clip-vit-base-patch32 is a much rougher copy (0.89 on
average), so onnxModel() warns when it is used; to get the close one,
convert the model yourself with exportOnnxModel("ViT-B-32"). A model you
have converted is used in place of the ready-made copy of the same name.
The patch tokens of an "int8" encoder (embedImages(tiles = 0)) are
less faithful than its image embeddings: with BioCLIP the lowest cosine
of a token with the "fp32" one was 0.86. Use tiles, or a higher
precision, to find sniglets with patch tokens.
dimoose's own layout, written by exportOnnxModel(): dimoose_onnx.json
(architecture, image size, preprocessing constants, file names),
image_encoder.onnx, text_encoder.onnx, their _fp16 and _int8
copies, and tokenizer.json. Its image encoder also returns the model's
patch tokens, so embedImages(tiles = 0) works.
The transformers.js layout of ready-made exports on the Hub: config.json,
preprocessor_config.json, tokenizer.json,
onnx/vision_model[_fp16|_quantized].onnx and
onnx/text_model[_fp16|_quantized].onnx. These give one embedding per
image and no patch tokens: tiles (the default of embedImages()) work,
tiles = 0 does not.
Could not download ...: the model is fetched from huggingface.co the
first time only. Files that arrived are kept, so calling again continues
where it stopped. A computer that cannot reach the Hub can be given the
model's folder, copied from onnxCache() on one that can, as name.
... is not a readable ONNX model: a model file is damaged, usually a download or copy that stopped early. Delete the file named in the message and load the model again; a file from the Hub is fetched afresh.
Could not load dimoose's ONNX code: the Python in use lacks
onnxruntime or tokenizers. The message gives the
reticulate::py_install() call that installs them.
exportOnnxModel() to convert a model, exportBrowserModel() to
use one in a web page, saveEmbeddings() to need no Python at all.
## Not run: model <- onnxModel() # CLIP ViT-B/32, downloaded once model <- onnxModel(precision = "fp16") # half the download images <- imageSet(system.file("extdata", "pics", package = "dimoose")) emb <- embedImages(model, images) exportOnnxModel("bioclip") # once, needs PyTorch model <- onnxModel("bioclip") # afterwards, without it ## End(Not run)## Not run: model <- onnxModel() # CLIP ViT-B/32, downloaded once model <- onnxModel(precision = "fp16") # half the download images <- imageSet(system.file("extdata", "pics", package = "dimoose")) emb <- embedImages(model, images) exportOnnxModel("bioclip") # once, needs PyTorch model <- onnxModel("bioclip") # afterwards, without it ## End(Not run)
The offline half of importFishbase(): parses a FishBase key description
page and its questions (couplet) page, both already downloaded.
parseFishbase( description, questions, separateTerms = TRUE, usePhyloService = c("GNA", "GNR", "none", "TNRS"), embedImages = FALSE, pageUrl = "https://www.fishbase.se/keys/questions.php" )parseFishbase( description, questions, separateTerms = TRUE, usePhyloService = c("GNA", "GNR", "none", "TNRS"), embedImages = FALSE, pageUrl = "https://www.fishbase.se/keys/questions.php" )
description |
The description page ( |
questions |
The questions page ( |
separateTerms |
If |
usePhyloService |
Taxonomic name resolution: |
embedImages |
If |
pageUrl |
URL of the questions page, used to resolve relative image links. |
A dimoose object.
## Not run: parseFishbase("description.html", "questions.html", usePhyloService = "none") ## End(Not run)## Not run: parseFishbase("description.html", "questions.html", usePhyloService = "none") ## End(Not run)
The lower-level half of keyFromText(): reads the text and returns one
row per lead together with everything that looked wrong, without building
a key. Use it to see what the parser made of a difficult text, fix the
text or the table, and then call keyFromLeads().
parseTextKey( text, llm = NULL, strict = FALSE, layout = c("auto", "bracketed", "indented", "unnumbered"), skip = NULL )parseTextKey( text, llm = NULL, strict = FALSE, layout = c("auto", "bracketed", "indented", "unnumbered"), skip = NULL )
text |
The key: a file path, one string holding the whole key, or a character vector of lines. Give the key alone, without the descriptions before and after it. |
llm |
Optional function for a first pass over messy text; see the section below. |
strict |
If |
layout |
|
skip |
Optional regular expression; lines that match are left out (for running heads the parser does not recognise by itself). |
A list of class textKeyParse with
leads: data frame with one row per lead: Statement (couplet),
Choice, Character (the text), Next (next couplet, or "-"),
Taxon, Authority, CommonName, Page and Line (the line the
lead starts on). Ready for keyFromLeads().
problems: data frame with Line, Couplet, Lead, Kind
("problem", "repair" or "note"), Type and Message.
title: the first line above the first lead, or NA.
layout: "bracketed", "indented", "paired" or "unnumbered".
text: the lines that were parsed (the model's answer when llm
returned text).
When llm is used, Line is the line of the source text each lead was
found on (NA if it could not be found) and leads$LlmLine is the
line or row of the model's answer.
# Primes for the second lead, tabs and "go to" p <- parseTextKey(c( "1. Shell coiled\tgo to 2", "1'. Shell of two valves\tBivalvia", "2. Shell with a lid\tLittorina, p. 40", "2'. Shell without a lid\tLymnaea" )) p$leads # A scanned page: what was repaired and what was skipped scan <- system.file("extdata", "textkeys", "meadow-ocr.txt", package = "dimoose") p <- parseTextKey(scan) p p$problems[p$problems$Kind == "repair", c("Line", "Message")] # An indented key without couplet numbers, as in older floras willows <- " Capsule glabrous. Leaves with close serrations, glaucous beneath. 11. S. melanopsis. Leaves entire or with distant serrations. Styles none. 12. S. exigua. Styles evident; aments dense. 16. S. sessilifolia. Capsule densely pubescent. 17. S. argophylla. " parseTextKey(willows)$leads[, c("Statement", "Choice", "Character", "Next", "Taxon")] # A stand-in for a language model that returns a table fake <- function(text) { data.frame( couplet = c(1, 1), lead = c("a", "b"), text = c("Wings present", "Wings absent"), destination = c("Pterygota", "Apterygota") ) } parseTextKey("1 Wings present ... Pterygota\n1 Wings absent ... Apterygota", llm = fake)$leads# Primes for the second lead, tabs and "go to" p <- parseTextKey(c( "1. Shell coiled\tgo to 2", "1'. Shell of two valves\tBivalvia", "2. Shell with a lid\tLittorina, p. 40", "2'. Shell without a lid\tLymnaea" )) p$leads # A scanned page: what was repaired and what was skipped scan <- system.file("extdata", "textkeys", "meadow-ocr.txt", package = "dimoose") p <- parseTextKey(scan) p p$problems[p$problems$Kind == "repair", c("Line", "Message")] # An indented key without couplet numbers, as in older floras willows <- " Capsule glabrous. Leaves with close serrations, glaucous beneath. 11. S. melanopsis. Leaves entire or with distant serrations. Styles none. 12. S. exigua. Styles evident; aments dense. 16. S. sessilifolia. Capsule densely pubescent. 17. S. argophylla. " parseTextKey(willows)$leads[, c("Statement", "Choice", "Character", "Next", "Taxon")] # A stand-in for a language model that returns a table fake <- function(text) { data.frame( couplet = c(1, 1), lead = c("a", "b"), text = c("Wings present", "Wings absent"), destination = c("Pterygota", "Apterygota") ) } parseTextKey("1 Wings present ... Pterygota\n1 Wings absent ... Apterygota", llm = fake)$leads
Cuts out the example regions of each sniglet and of each lead, stores them
as PNGs in the key's features and meta, and points the leads at them
so exportWizard() shows them. A "Has" lead shows the sniglet in the
images that take that lead; a "Lacks" lead shows what its images have
instead (see keyFromSniglets()). The glossary shows each sniglet's
examples across all the images.
patchExemplars(model, key, images, pad = NULL, size = 96)patchExemplars(model, key, images, pad = NULL, size = 96)
model |
|
key |
A key from |
images |
The |
pad |
Pixels of context to show around the region. By default none for tiles, and half a patch's width for patch tokens. |
size |
The crop's output size, px. |
The key, modified in place and returned invisibly.
The 5,435 haplogroups of PhyloTree Build 17 (van Oven 2015) as
distributed by Haplogrep 3 (release 17.2 of phylotree-rsrs-17), one row
per haplogroup, rooted at mtMRCA. Use phylotreeKey() to turn the tree,
or part of it, into an identification key, and phylotree17_mutations for
the mutations one per row.
phylotree17phylotree17
A data frame with 5,435 rows and 5 columns:
Haplogroup name.
Name of the parent haplogroup (NA for mtMRCA).
Number of branches from mtMRCA.
Number of direct subclades.
Space-separated mutations of the branch leading to this haplogroup, as written by Haplogrep.
Rows are in the order of Haplogrep's tree.xml, which is preorder (each
haplogroup directly before its subclades). Names and mutations are
Haplogrep's, verbatim: names such as H2a+152 16311 or K1b1+(16093)
label branches that PhyloTree leaves unnamed, and two names (M4’’67,
M4’’67+16311!) contain the Unicode character U+2019. Haplogrep's
trailing space on one name is trimmed.
Mutations are in RSRS orientation: each is the state on the branch leading
to the haplogroup. 16362C is a substitution (position and derived base),
263G! a back-mutation (it reverts the latest change at that position;
!! marks a second reversion), 315.1C or 5899.XC an insertion (X:
unspecified length) and 249d a deletion. Unlike some other copies (e.g.
MToolBox's), PhyloTree's unstable mutations, which PhyloTree prints in
parentheses, are included; the parentheses are not recorded.
Haplogrep 3 tree phylotree-rsrs-17, release 17.2, commit
0e79ddf1b78f7a1b81fb5d607bc31a6ddea7b18b, src/tree.xml
(https://github.com/genepi/phylotree-rsrs-17), MIT license. The
conversion and a script that checks these data against GitHub are in
data-raw/phylotree17/ in the package sources.
van Oven M (2015). PhyloTree Build 17: Growing the human mitochondrial DNA tree. Forensic Science International: Genetics Supplement Series 5, e392-e394. doi:10.1016/j.fsigss.2015.09.155
Schönherr S, Weissensteiner H, Kronenberg F, Forer L (2023). Haplogrep 3 - an interactive haplogroup classification and analysis platform. Nucleic Acids Research 51, W263-W268. doi:10.1093/nar/gkad284
phylotree17_mutations, phylotreeKey()
head(phylotree17) phylotree17[phylotree17$parent %in% "H2a2a", ]head(phylotree17) phylotree17[phylotree17$parent %in% "H2a2a", ]
The mutations of every haplogroup in phylotree17, one row per mutation, in the order Haplogrep lists them, with the bases before and after worked out by following the tree from the RSRS.
phylotree17_mutationsphylotree17_mutations
A data frame with 13,384 rows and 6 columns:
Haplogroup whose branch carries the mutation.
The mutation as written by Haplogrep.
Position in the RSRS/rCRS coordinates.
transition, transversion, back-mutation,
insertion or deletion.
Base before the mutation (for a reverted insertion, the
inserted sequence); NA for insertions.
Base after the mutation, or the inserted sequence; NA
for deletions and reverted insertions.
Seven back-mutations revert a position that nothing earlier on their path
changes (six 16233C!, e.g. in D4c1b1, and D5a2 1438A!); for those
the written base is taken as the result.
See phylotree17.
table(phylotree17_mutations$type) phylotree17_mutations[phylotree17_mutations$haplogroup == "J1c", ]table(phylotree17_mutations$type) phylotree17_mutations[phylotree17_mutations$haplogroup == "J1c", ]
Turns phylotree17, or the part of it below root, into a dimoose key.
Each haplogroup with subclades becomes a step at which the user picks the
subclade whose defining mutations the sample carries. With paragroups = TRUE every step also offers "none of these", which identifies the
paragroup, written as in PhyloTree with a trailing * (e.g. H2a*).
phylotreeKey(root = "mtMRCA", paragroups = TRUE, maxDepth = Inf)phylotreeKey(root = "mtMRCA", paragroups = TRUE, maxDepth = Inf)
root |
Haplogroup to start from; the default, |
paragroups |
If |
maxDepth |
Maximum number of steps below |
The whole tree gives a very large key (2,420 steps and 5,435 possible
results with paragroups), so for exportWizard() a subtree or a
maxDepth is usually more practical.
A dimoose object whose leads have a Label column with the
haplogroup names.
h2a <- phylotreeKey("H2a") head(h2a$leads) ## Not run: exportWizard(phylotreeKey("U5"), "u5.html") exportWizard(phylotreeKey(maxDepth = 6), "macrohaplogroups.html") ## End(Not run)h2a <- phylotreeKey("H2a") head(h2a$leads) ## Not run: exportWizard(phylotreeKey("U5"), "u5.html") exportWizard(phylotreeKey(maxDepth = 6), "macrohaplogroups.html") ## End(Not run)
Keeps only the taxa you name (or removes the ones you name) and tidies up what is left: leads that no longer reach any kept taxon are removed, and a couplet that no longer splits anything is taken out, with the lead that led to it now leading straight to where its one remaining lead went. Collapses are repeated until every couplet again has at least two leads going to different places, so chains of couplets, couplets reached from several leads and the first couplet of the key are all handled.
pruneKey( key, keep = NULL, drop = NULL, ranks = c("genus", "exact"), note = c("column", "append", "none"), renumber = FALSE )pruneKey( key, keep = NULL, drop = NULL, ranks = c("genus", "exact"), note = c("column", "append", "none"), renumber = FALSE )
key |
A dimoose key. |
keep |
Taxa to keep; all others are removed. |
drop |
Taxa to remove (after |
ranks |
|
note |
|
renumber |
If |
The leads that remain keep their text, their order and (unless
renumber = TRUE) their couplet names, so every kept taxon is reached by
the same leads as before, less the ones that no longer decide anything.
Pruning in two steps gives the same key as pruning once to the taxa kept
by both.
A dimoose key with the same columns in its leads as key
(plus Note if a note was written). Embedded images, taxa and features
that no remaining lead uses are left out. It is an error if no result is
left, or only one (a key needs two results to tell apart).
Names are compared without regard to case or extra spaces, and the
multiplication sign of a hybrid may be typed as a lower-case x
("Populus x acuminata"). An entry of keep or drop matches a taxon of
the key when it equals the taxon's whole label
("Bur oak (Quercus macrocarpa)"), the scientific name in its
parentheses ("Quercus macrocarpa") or the common name before them
("Bur oak"); entries may themselves be written with or without a common
name.
With ranks = "genus" (the default) a name also stands for what lies
below it, judged by the words the names begin with:
a genus in keep or drop ("Populus") matches every species of that
genus in the key, and a species ("Populus balsamifera") matches its
subspecies and varieties ("Populus balsamifera subsp. trichocarpa");
a name matches the same name followed by its author
("Quercus macrocarpa" and "Quercus macrocarpa Michx.", or
"Carcharodon carcharias (Linnaeus, 1758)"), whichever of the two has
the author;
a species in keep ("Quercus macrocarpa") keeps a result of the key
that is only its genus ("Quercus"), and a subspecies keeps its
species. This is not done for drop: removing one species does not
remove its genus.
With ranks = "exact" only whole labels, scientific names and common
names match.
Families and higher ranks cannot be told from a name, so a result such as
"Salicaceae" is matched only by itself, and a synonym is a different
name; regionalKey() asks GBIF or iNaturalist instead and so works at any
rank and through synonyms.
A result that names several taxa the key cannot tell apart, written
"A / B" as keyFromRpart() and keyFromMatrix() write them, is treated
as the taxa A and B: each can be kept or dropped by its own name, and
when only A is kept the result becomes "A". Naming the whole result
("A / B") keeps or drops all of it. (A label with a common name, such as
"Aspen / Popple (Populus tremuloides)", is one taxon.)
When a couplet is taken out, the text of its one remaining lead still
describes the taxa beyond it, and is useful for confirming an
identification. With note = "column" (the default) it is added to a
Note column on the lead that now skips the couplet; with
note = "append" it is added to that lead's Character in square
brackets, so it shows wherever the key is displayed; with note = "none"
it is discarded. Whatever note is, the removed lead's pictures (Image
and ImageLink) are added to those of the lead that now skips the
couplet. When the first couplet of the key is taken out there is no lead
to carry the text: it goes in the key's metadata as pruned_note (added
to any that is already there), and its pictures are left out.
regionalKey() to prune to the taxa recorded in a place,
mergeKeys() to go the other way.
cottonwoods <- pruneKey(montanaKey(), keep = "Populus") cottonwoods cottonwoods$leads[, c("Statement", "Choice", "Next", "Taxon")] # names can be given with or without the common name k <- pruneKey(montanaKey(), drop = c("Acer negundo", "Bur oak (Quercus macrocarpa)"), renumber = TRUE) k$validate() k$leads$Notecottonwoods <- pruneKey(montanaKey(), keep = "Populus") cottonwoods cottonwoods$leads[, c("Statement", "Choice", "Next", "Taxon")] # names can be given with or without the common name k <- pruneKey(montanaKey(), drop = c("Acer negundo", "Bur oak (Quercus macrocarpa)"), renumber = TRUE) k$validate() k$leads$Note
An R6 object holding a FishBase key's description and its couplet table as downloaded, before conversion to a dimoose object.
descKey title (read-only).
dfThe key's couplet table: one row per lead, with columns
Statement, Choice, Character, Next, Prev and Taxon.
rawhtml$print()Print the description, table and metadata.
rawhtml$print(...)
...Unused.
rawhtml$new()Download a FishBase key.
rawhtml$new( keycode, fishbaseUrl = "https://www.fishbase.se/", separateTerms = TRUE )
keycodeFishBase key code.
fishbaseUrlBase URL of the FishBase mirror.
separateTermsIgnored; kept for backward compatibility.
rawhtml$clone()The objects of this class are cloneable with this method.
rawhtml$clone(deep = FALSE)
deepWhether to make a deep clone.
DELTA (DEscription Language for TAxonomy, Dallwitz 1980) is the oldest
and still widely used text format for descriptive data, and a TDWG
standard. A dataset is classically three files: specs (how many
characters there are and of which type), chars (the character list with
the states) and items (the taxa, each with its attributes).
readDELTA() reads them into a character matrix (see readSDD() for the
convention), writeDELTA() writes a matrix as the three files.
readDELTA( specs, chars = NULL, items = NULL, taxon = "Taxon", sep = "/", inapplicable = NA, comments = FALSE ) writeDELTA( x, dir, characters = NULL, taxon = "Taxon", sep = "/", inapplicable = NULL, title = NULL, files = c(specs = "specs", chars = "chars", items = "items") )readDELTA( specs, chars = NULL, items = NULL, taxon = "Taxon", sep = "/", inapplicable = NA, comments = FALSE ) writeDELTA( x, dir, characters = NULL, taxon = "Taxon", sep = "/", inapplicable = NULL, title = NULL, files = c(specs = "specs", chars = "chars", items = "items") )
specs |
Path of the |
chars, items
|
Paths of the |
taxon |
Name of the taxon column of the matrix. |
sep |
Separator between the states of a variable taxon in one cell. |
inapplicable |
For readers, the cell value to give "not applicable"
(default |
comments |
If |
x |
A character matrix (data frame), or the list returned by
|
dir |
Folder to write the three files into; created if needed. |
characters |
Optional data frame describing the characters
( |
title |
Title put in the |
files |
Names of the three files. |
readDELTA() returns a tdwgData list like readSDD(), with
matrix, characters (Character, Type, States, Unit,
InapplicableIf, Number), taxa (Taxon, Comment) and meta.
writeDELTA() returns the paths of the three files, invisibly.
The directives *CHARACTER LIST, *ITEM DESCRIPTIONS,
*CHARACTER TYPES, *NUMBERS OF STATES, *NUMBER OF CHARACTERS,
*IMPLICIT VALUES and *DEPENDENT CHARACTERS, wherever in the given
files they are (so a dataset kept in one file, or split differently, reads
too); other directives are skipped. Character types are unordered
multistate (UM, read as "categorical"), ordered multistate (OM,
"ordered"), integer (IN, "integer"), real (RN, "real") and text
(TE, "text").
Attributes follow the DELTA standard: 12,1/2 (state 1 or 2) and 12,1&2
(1 and 2) both give "state 1/state 2", so the difference between "or"
and "and" is not kept; 13,2-3 is a range, which for
an ordered character includes the states in between; 14,U is unknown
(NA), 15,- not applicable (inapplicable), 16,V variable (all
states). Numeric values are kept as written, e.g. "7-8.5" or
"(1-)2-3(-4)", which is the numeric notation of the matrix convention.
For a text character the cell is the text of the comment (7<possibly two species>). Implicit values are filled in. A dependent character that is
not coded is "not applicable" when its controlling character only takes
the controlling states; the rule itself is kept in the InapplicableIf
column of characters. Variant items (#+) take the attributes they
leave out from their main item, as the standard says.
Comments (text in angle brackets) are dropped from the names of
characters, states and taxa unless comments = TRUE, and from the
attributes of non-text characters; the comment of a taxon name (usually
the authority) is kept in taxa$Comment. A name that consists of nothing
but a comment (#4. <longevity of plants>/) is that comment. Typesetting
marks (the RTF control words DELTA programs use, such as \i{}, \b0{}
or \par{}) are removed, and those that stand for a character
(\endash{}, \ldblquote{}) become that character. Files that are not
valid UTF-8 are read as Windows-1252. The obsolete line sequence numbers
are not supported.
The reader was checked against the datasets distributed with Open DELTA (among them the "Grass Genera" sample, a flora of 506 taxa and a virus database of 969 taxa) by comparing every coded attribute with a second, independent parse of the files.
specs gets the numbers of characters, states and items, the character
types, the numbers of states and the dependent characters; chars the
character list; items one item per taxon. Every coded cell is written
out (no implicit values are used), NA cells are left out, which DELTA
reads as unknown, and cells equal to inapplicable are written as -.
The states of a variable taxon are joined with / ("or"). A numeric cell
that is not a DELTA number or range, and all cells of text characters,
are written as comments. Encoding is UTF-8, and lines are wrapped at 78
columns wherever they have a blank to break at.
A few things cannot be written as they are, because DELTA would read
them as syntax: angle brackets in a name make a comment (which
readDELTA() drops unless comments = TRUE); a star or, in a name, a
numero after a blank is put in parentheses ((*), (#)); the blank
after a slash in a name is dropped. No file written by dimoose has been
read into a DELTA program (Confor, Intkey, Open DELTA).
Dallwitz, M. J. & Paine, T. A. (1992 onwards). Definition of the DELTA format. https://www.delta-intkey.com/www/standard.htm
dir <- system.file("extdata", "tdwg", "beetles-delta", package = "dimoose") d <- readDELTA(dir) d d$matrix d$characters[, c("Character", "Type", "States", "Unit")] out <- writeDELTA(d, tempfile("delta")) out identical(readDELTA(dirname(out[1]))$matrix, d$matrix)dir <- system.file("extdata", "tdwg", "beetles-delta", package = "dimoose") d <- readDELTA(dir) d d$matrix d$characters[, c("Character", "Type", "States", "Unit")] out <- writeDELTA(d, tempfile("delta")) out identical(readDELTA(dirname(out[1]))$matrix, d$matrix)
Parses the nested <haplogroup> XML that Haplogrep distributes (for
example phylotree-rcrs-17's tree.xml) into the same data frame shape as
phylotree17, so any Haplogrep tree can be loaded, not just the bundled
snapshot. Each haplogroup's <details><poly> entries become its mutations.
readHaplogrep(file, key = FALSE)readHaplogrep(file, key = FALSE)
file |
Path to the tree XML. |
key |
If |
A data frame with haplogroup, parent, depth, n_subclades
and mutations, or a dimoose object if key = TRUE.
## Not run: tree <- readHaplogrep("tree.xml") ## End(Not run)## Not run: tree <- readHaplogrep("tree.xml") ## End(Not run)
Lucid (Identic / Lucidcentral) is a widely used commercial program for
interactive matrix keys: "entities" (taxa) scored for the states of
"features" (characters). readLucid() reads a key exported from the Lucid
Builder into a character matrix (see readSDD() for the convention) and
writeLucid() writes a matrix in a form the Builder imports.
readLucid( file, format = NULL, taxon = "Taxon", sep = "/", inapplicable = NA, entities = c("columns", "rows"), scores = c(1, 2, 3), fullNames = FALSE ) writeLucid( x, file, format = c("csv", "sdd"), characters = NULL, taxon = "Taxon", sep = "/", inapplicable = NULL, entities = c("columns", "rows"), title = NULL )readLucid( file, format = NULL, taxon = "Taxon", sep = "/", inapplicable = NA, entities = c("columns", "rows"), scores = c(1, 2, 3), fullNames = FALSE ) writeLucid( x, file, format = c("csv", "sdd"), characters = NULL, taxon = "Taxon", sep = "/", inapplicable = NULL, entities = c("columns", "rows"), title = NULL )
file |
Path of the file to read or write. |
format |
|
taxon |
Name of the taxon column of the matrix. |
sep |
Separator between the states of a variable taxon in one cell. |
inapplicable |
For readers, the cell value to give "not applicable"
(default |
entities |
For CSV: whether the entities (taxa) are the |
scores |
Which Lucid scores mean that the taxon has the state. |
fullNames |
If |
x |
A character matrix (data frame), or the list returned by
|
characters |
Optional data frame describing the characters
( |
title |
Title of the dataset (SDD only). |
readLucid() returns a tdwgData list like readSDD(): matrix,
characters (Character, Type, States, Unit, Path), taxa
(Taxon, Path) and meta. writeLucid() returns file, invisibly.
Lucid has several export formats, and they are not equally well documented:
"csv" (Lucid Builder: File > Export Key and Import Key,
file type CSV). Documented in the Lucid Builder help, which also offers
a small example key with its CSV export: the first row holds the
entities, every further row one state named Feature:State (a
colon-separated path through the feature tree), and the cells hold the
scores 0 absent, 1 present, 2 rarely present, 3 uncertain, 4 present by
misinterpretation, 5 rarely present by misinterpretation, 6 not scoped.
A numeric feature is one row named Feature# with cells
score type:rare low:low:high:rare high, or a plain 0 when it is not
scored. This is the format writeLucid() writes by default, and the
documented way to get a matrix into Lucid. Use entities = "rows"
for the Builder's "transposed" variant.
"sdd": Lucid exports matrix keys as SDD, the TDWG standard; the
file is read with readSDD() and written with writeSDD().
"lif3" (Lucid Interchange Format 3, .lif3): an XML file. Lucid
publishes no schema or description of it beyond saying that it is XML,
so readLucid() reads it as far as real exported files show how:
features, states and entities (<item>), the feature and entity trees,
and the multistate scores. This was checked against four published keys
(Lucid 3 and Lucid 4 exports), one of which was also published as SDD:
the two readings agree in all 3454 scores. Those files only use the
scores 1 (present) and 3 (uncertain); the scores 2, 4, 5 and 6 are taken
to mean what they mean in CSV, as they do in the Builder's own key
files, which store scores in the same elements. No real LIF3 file with
numeric scores was found: they are read when they are laid out as in the
Builder's own key files (<scored_data> with omin, nmin, nmax,
omax), and otherwise left out with a warning. Units, dependencies,
text scores, subsets and media of a LIF3 file are not read (a warning
says so when the file has dependencies or text scores), and dimoose does
not write LIF3; export CSV or SDD from the Builder when you need those.
LIF (Lucid 2, .lif, a sectioned text file) and the Builder's
"XML Dataset" are not publicly specified; readLucid() stops and
asks for a CSV or SDD export instead of guessing.
By default a taxon gets a state when it is scored present (1), rarely
present (2) or uncertain (3) for it; states scored only "by
misinterpretation" are left out, because the taxon does not really have
them. A feature for which every state of a taxon is uncertain, or nothing
is scored, gives NA; "not scoped" gives inapplicable. writeLucid()
writes 1 for every state in a cell and 0 otherwise, and 6 for cells equal
to inapplicable.
For numeric features the Lucid help gives the layout of the cell but not
the values its score type can take. The one real export at hand uses 1
for a scored range and a plain 0 for a feature that is not scored, and
that is all writeLucid() writes: 1:rare low:low:high:rare high for a
value or range, 0 for NA. A numeric cell equal to inapplicable is
also written as 0, with a warning, because no score for "not scoped" is
documented for numeric features. On reading, type 1 gives the range, 0
gives NA, 6 is taken as "not scoped" by analogy with the multistate
scores, and any other type gives NA with a warning.
Text characters cannot be stored in a Lucid matrix and are left out with a
warning. Colons in names are replaced, because Lucid reads them as steps
of a path; data that was read from Lucid keeps its paths (the Path
columns of characters and taxa). Units of numeric features are not
part of the CSV format. No file written by dimoose has been imported
into the Lucid Builder.
Lucid Builder help, "CSV (Comma Separated Value) Files" and "Importing and Exporting", https://help.lucidcentral.org/lucid/.
readLucidKey() for Lucid pathway (dichotomous) keys,
readSDD(), readDELTA().
f <- system.file("extdata", "tdwg", "bears-lucid.csv", package = "dimoose") d <- readLucid(f) d$matrix d$characters out <- writeLucid(d, tempfile(fileext = ".csv")) readLines(out) identical(readLucid(out)$matrix, d$matrix)f <- system.file("extdata", "tdwg", "bears-lucid.csv", package = "dimoose") d <- readLucid(f) d$matrix d$characters out <- writeLucid(d, tempfile(fileext = ".csv")) readLines(out) identical(readLucid(out)$matrix, d$matrix)
The Lucid Builder also makes "pathway" keys: ordinary dichotomous keys of
couplets and leads (the successor of the stand-alone Lucid Phoenix).
readLucidKey() reads one of its exports into a dimoose key and
writeLucidKey() writes a dimoose key in the same forms. All four are
described in the Lucid Builder help ("Pathway key exports"):
readLucidKey(file, format = NULL, desc = NULL) writeLucidKey(key, file, format = c("lpxk", "csv", "text", "sdd"))readLucidKey(file, format = NULL, desc = NULL) writeLucidKey(key, file, format = c("lpxk", "csv", "text", "sdd"))
file |
Path of the file to read or write. |
format |
|
desc |
Title of the key; by default the title in the file, if any. |
key |
A dimoose key. |
"lpxk", Lucid Phoenix XML: <PhoenixKey> with the taxa as
<Identity> elements and the couplets as <Step> elements holding
<Lead>s whose goto names the next step or the taxon. The step's
text becomes the couplet's Question, an identity's url the
TaxonUrl. Filters (<FilterItem>) and icons are not read or written.
"csv": one line per lead, couplet,"lead text",next couplet or taxon.
"text": the same with blanks instead of commas, as printed in a
paper.
"sdd": SDD with an identification key; see readSDDKey().
In the CSV and text forms the last field is read as a couplet when it
matches a couplet number and as a taxon otherwise, so writeLucidKey()
refuses to write a key in which a taxon is named like a couplet. These
two forms hold neither the title of the key nor questions, links or
images. The CSV, text and Phoenix XML forms follow the examples printed
in the Lucid help; no file exported by the Lucid Builder itself was
available to check them against, and no file written by dimoose has been
imported into the Builder.
readLucidKey() returns a dimoose key; writeLucidKey()
returns file, invisibly.
Lucid Builder help, "Pathway key exports", https://help.lucidcentral.org/lucid/pathway-key-exports/.
readLucid() for Lucid matrix keys, readSDDKey(),
keyFromSpreadsheet().
f <- system.file("extdata", "tdwg", "insects.lpxk", package = "dimoose") k <- readLucidKey(f) k k$leads[, c("Statement", "Choice", "Character", "Next", "Taxon")] out <- writeLucidKey(k, tempfile(fileext = ".csv"), format = "csv") readLines(out)f <- system.file("extdata", "tdwg", "insects.lpxk", package = "dimoose") k <- readLucidKey(f) k k$leads[, c("Statement", "Choice", "Character", "Next", "Taxon")] out <- writeLucidKey(k, tempfile(fileext = ".csv"), format = "csv") readLines(out)
SDD is the XML standard of TDWG (Biodiversity Information Standards) for
descriptive data: characters and their states, the taxa, and how each
taxon is coded for each character. readSDD() turns an SDD file into a
character matrix, writeSDD() writes one. Both follow SDD 1.1 (the
http://rs.tdwg.org/UBIF/2006/ schema), which is what Lucid, Xper2/Xper3
and DELTA tools export.
readSDD(file, taxon = "Taxon", sep = "/", inapplicable = NA, dataset = NULL) writeSDD( x, file, characters = NULL, taxon = "Taxon", title = NULL, sep = "/", inapplicable = NULL, lang = "en", key = NULL )readSDD(file, taxon = "Taxon", sep = "/", inapplicable = NA, dataset = NULL) writeSDD( x, file, characters = NULL, taxon = "Taxon", title = NULL, sep = "/", inapplicable = NULL, lang = "en", key = NULL )
file |
Path of the SDD file to read or write. |
taxon |
Name of the taxon column of the matrix. |
sep |
Separator between the states of a variable taxon in one cell. |
inapplicable |
For readers, the cell value to give "not applicable"
(default |
dataset |
Which |
x |
A character matrix (data frame), or the list returned by
|
characters |
Optional data frame describing the characters: a
|
title |
Title of the dataset; by default the |
lang |
Language of the labels, as an |
key |
Optional dimoose key to store in the same file as an SDD
identification key (see |
readSDD() returns a list of class tdwgData with
matrix: the character matrix;
characters: one row per character with Character (the column
name), Type ("categorical", "ordered", "integer", "real" or
"text"; SDD itself only tells categorical, real and text apart),
States (all defined states, separated by sep), Unit,
InapplicableIf ("Other character=state/state": the character does
not apply when the other character has one of these states; several
rules are separated by ;), Cost when the file has ratings, and Id;
taxa: Taxon, Detail and Id;
meta: key/value data frame (title, description, language,
authors, copyright, generator, created, and keys, the number of
identification keys in the file, see readSDDKey()).
writeSDD() returns file, invisibly.
A plain data frame with one row per taxon, the taxon names in a column
(Taxon by default) and one column per character. Cells are text:
a state name, or several separated by sep when the taxon is variable
("red/yellow");
for numeric characters a value or range written the way DELTA and floras
write it: "5", "2.3-7.9", "2.3-5.1-7.9" (the middle value is the
mean or median) and, with rare extremes, "(1-)2-8(-12)";
for text characters the text;
NA when the character is unknown, not coded or not applicable (give
inapplicable to tell "not applicable" apart).
Categorical, quantitative and text characters, in the order of the
<Characters> list; the coded descriptions (<SummaryData>) of each
taxon; <Status> codes (NotApplicable becomes inapplicable, the
others NA); the dependency rules of the character tree (InapplicableIf,
OnlyApplicableIf); and observation-convenience ratings, which become a
Cost from 1 (easy) to 5. A taxon with several descriptions gets one row,
with the states of all of them. States scored with a "misinterpretation"
modifier (Lucid's way of catching a common mistake) are not taken as
states of the taxon, and states scored with the modifier the Lucid Builder
labels "unscoped" make the cell "not applicable" (the Builder defines that
modifier in every SDD file it writes; none of the real exports dimoose
was checked against uses it, so how it is attached to a score is assumed
to be the same as for the other modifiers). Other modifiers (rarely,
uncertain) are ignored and the state kept, so a character whose states
are all "uncertain" for a taxon lists all of them. Dependency rules are
read from the character itself and from the groups of the character tree
it sits in. Quantitative measures map to the range text as follows:
Min/Max are the ends of the range, unless UMethLower/UMethUpper
are given too, in which case those are the normal range and Min/Max
the extremes in parentheses; Mean (or Median, Mode) is the middle
value. Other statistics (SD, N) are dropped.
Not read: sample data (<SampleData> of individual specimens), media,
natural-language descriptions, taxon hierarchies, and text in more than
one language (the first label of each object is used).
A complete, schema-valid SDD 1.1 document with the taxon names, the
characters, a character tree holding the dependency rules, and one coded
description per taxon. NA cells are left out (SDD reads a missing
coding as unknown), cells equal to inapplicable get the status
NotApplicable. A Cost from 1 to 5 is written as an
observation-convenience rating, as Xper3 does. "ordered" characters are
written as categorical ones and "integer" as quantitative ones, so those
two types come back as "categorical" and "real". Of a cell listing
several numeric ranges ("5/10-15") the overall range is written, and
numbers are written as numbers ("0.40" comes back as "0.4"). A
categorical character for which no taxon has a state is written as a text
character, with a warning, because SDD does not allow a categorical
character without states. Files written from the real datasets dimoose
was tried on (DELTA, Lucid and Xper exports of up to 969 taxa) all
validate against the SDD 1.1 schema; none has been imported into Lucid
or Xper.
Hagedorn, G., Thiele, K., Morris, R. & Heidorn, P. B. (2005). Structured Descriptive Data (SDD) w3c-xml-schema, version 1.0. https://www.tdwg.org/standards/sdd/
readSDDKey() for dichotomous keys stored in SDD, readXper(),
readLucid(), readDELTA().
f <- system.file("extdata", "tdwg", "violets.sdd.xml", package = "dimoose") d <- readSDD(f) d d$matrix d$characters[, c("Character", "Type", "States", "Unit")] # write a matrix of your own m <- data.frame( Taxon = c("Apis mellifera", "Bombus terrestris", "Vespula vulgaris"), Hair = c("sparse", "dense", "sparse/absent"), "Body length" = c("12-15", "(11-)14-17(-22)", NA), check.names = FALSE ) chars <- data.frame(Character = "Body length", Type = "real", Unit = "mm") out <- writeSDD(m, tempfile(fileext = ".xml"), characters = chars, title = "Three wasps and bees") readSDD(out)$matrixf <- system.file("extdata", "tdwg", "violets.sdd.xml", package = "dimoose") d <- readSDD(f) d d$matrix d$characters[, c("Character", "Type", "States", "Unit")] # write a matrix of your own m <- data.frame( Taxon = c("Apis mellifera", "Bombus terrestris", "Vespula vulgaris"), Hair = c("sparse", "dense", "sparse/absent"), "Body length" = c("12-15", "(11-)14-17(-22)", NA), check.names = FALSE ) chars <- data.frame(Character = "Body length", Type = "real", Unit = "mm") out <- writeSDD(m, tempfile(fileext = ".xml"), characters = chars, title = "Three wasps and bees") readSDD(out)$matrix
Besides character matrices, an SDD file can hold ready-made
identification keys (<IdentificationKey> elements): a tree of leads,
each with a statement and either further leads, a taxon or a pointer to
another key in the file. readSDDKey() reads one into a dimoose key,
writeSDDKey() stores a dimoose key that way. Lucid's pathway-key
builder exports this form.
readSDDKey(file, key = 1, desc = NULL, dataset = NULL) writeSDDKey(key, file, title = NULL, lang = "en")readSDDKey(file, key = 1, desc = NULL, dataset = NULL) writeSDDKey(key, file, title = NULL, lang = "en")
file |
Path of the SDD file. |
key |
For |
desc |
Title of the key; by default the key's label in the file. |
dataset |
Which |
title |
Title of the SDD dataset; by default the key's description. |
lang |
Language of the text, as an |
In SDD the leads form a tree (each lead names its parent lead) and are
not numbered. On reading, the leads that share a parent become a couplet
and the couplets are numbered in the usual order of a printed key (the
first lead of couplet 1 goes to couplet 2, and so on). A lead that points
to another key of the same file (<Subkey>) continues with that key's
first couplet, so a key split over several SDD keys comes back as one. A
question on a lead (or on the key itself) becomes the Question of the
couplet it leads to. A lead that ends without a taxon is kept, and
reported by key$validate().
On writing, a couplet that is reached from more than one lead
(a reticulate key) is written out once under each of them, because an SDD
lead can have only one parent. Images and links attached to leads or taxa
(Image, TaxonUrl) and other extra columns of the key are not written.
readSDDKey() returns a dimoose key; writeSDDKey() returns
file, invisibly.
readSDD() for the character matrix of an SDD file,
readLucidKey() for Lucid's own pathway-key formats.
f <- system.file("extdata", "tdwg", "insects-key.sdd.xml", package = "dimoose") k <- readSDDKey(f) k k$leads[, c("Statement", "Choice", "Character", "Next", "Taxon")] out <- writeSDDKey(sharkKey(), tempfile(fileext = ".xml")) readSDDKey(out)f <- system.file("extdata", "tdwg", "insects-key.sdd.xml", package = "dimoose") k <- readSDDKey(f) k k$leads[, c("Statement", "Choice", "Character", "Next", "Taxon")] out <- writeSDDKey(sharkKey(), tempfile(fileext = ".xml")) readSDDKey(out)
Xper3 (and its desktop predecessor Xper2) keep a knowledge base of taxa
("items") described by characters ("descriptors"). Their documented
interchange format is SDD: export the knowledge base as SDD from Xper, and
import an SDD file written by writeXper() there. Xper's own storage
format is not publicly specified, so dimoose does not try to read or write
it; these two functions read and write the SDD dialect that Xper
produces.
readXper(file, taxon = "Taxon", sep = "/", inapplicable = NA, strict = TRUE) writeXper( x, file, characters = NULL, taxon = "Taxon", title = NULL, sep = "/", inapplicable = NULL, lang = "en" )readXper(file, taxon = "Taxon", sep = "/", inapplicable = NA, strict = TRUE) writeXper( x, file, characters = NULL, taxon = "Taxon", title = NULL, sep = "/", inapplicable = NULL, lang = "en" )
file |
Path of the SDD file to read or write. |
taxon |
Name of the taxon column of the matrix. |
sep |
Separator between the states of a variable taxon in one cell. |
inapplicable |
For readers, the cell value to give "not applicable"
(default |
strict |
If |
x |
A character matrix (data frame), or the list returned by
|
characters |
Optional data frame describing the characters: a
|
title |
Title of the dataset; by default the |
lang |
Language of the labels, as an |
Compared with plain readSDD(), the Xper dialect has a few habits that
are handled here: Xper2 names each taxon in the label of its coded
description rather than in a taxon list; the dependencies between
characters are InapplicableIf rules in a single character tree; and
every coding carries an observation-convenience rating, which is read as
the character's Cost (1 = easy to 5 = hard). readXper() stops if the
file was not written by Xper, unless strict = FALSE.
readXper() returns the same tdwgData list as readSDD();
writeXper() returns file, invisibly.
Vignes-Lebbe, R., Chesselet, P. & Diep Thi, M.-H. (2016). Xper3: new tools for collaborating, training and transmitting knowledge on botanical phenotypes. In: Botanists of the twenty-first century, UNESCO. https://www.xper3.fr/
f <- system.file("extdata", "tdwg", "felids-xper3.sdd.xml", package = "dimoose") d <- readXper(f) d$matrix d$characters[, c("Character", "InapplicableIf", "Cost")] out <- writeXper(d, tempfile(fileext = ".sdd.xml")) identical(readXper(out)$matrix, d$matrix)f <- system.file("extdata", "tdwg", "felids-xper3.sdd.xml", package = "dimoose") d <- readXper(f) d$matrix d$characters[, c("Character", "InapplicableIf", "Cost")] out <- writeXper(d, tempfile(fileext = ".sdd.xml")) identical(readXper(out)$matrix, d$matrix)
Sites that mutate independently on many branches of the tree, such as 152, 16311 and 16189 in PhyloTree. These are the variants whose effects a tree can find across unrelated lineages.
recurrentPositions( min_branches = 10, types = c("transition", "transversion", "back-mutation"), mutations = dimoose::phylotree17_mutations )recurrentPositions( min_branches = 10, types = c("transition", "transversion", "back-mutation"), mutations = dimoose::phylotree17_mutations )
min_branches |
Minimum number of branches (haplogroups) on which the site mutates. |
types |
Mutation types to count; by default substitutions and
back-mutations. Insertions and deletions are left out, except that a
reverted insertion counts as a back-mutation at its insertion site
(for example |
mutations |
Default phylotree17_mutations. |
A data frame of site and n_branches, most recurrent first.
head(recurrentPositions())head(recurrentPositions())
Asks GBIF or iNaturalist
which of the key's taxa have been recorded in a place and prunes the key
to those with pruneKey(): the key to the trees of Flathead County from
a key to the trees of Montana, in one line.
regionalKey( key, place, source = c("gbif", "inat"), minRecords = 1, kingdom = NULL, unknown = c("keep", "drop"), filters = list(), ..., fetch = prnFetch )regionalKey( key, place, source = c("gbif", "inat"), minRecords = 1, kingdom = NULL, unknown = c("keep", "drop"), filters = list(), ..., fetch = prnFetch )
key |
A dimoose key. |
place |
The place; see the section for the source used. |
source |
|
minRecords |
Keep a taxon with at least this many records. |
kingdom |
For GBIF, the kingdom of the key's taxa ( |
unknown |
What to do with results the source does not know by name
(such as |
filters |
Named list of further search parameters for the count:
for GBIF, occurrence search parameters such as
|
... |
Passed to |
fetch |
Function that makes a request: called as |
Each result of the key is looked up by the scientific name in its
parentheses ("Bur oak (Quercus macrocarpa)"), or by the whole result
when there is none; the taxa of a shared result ("A / B") are looked up
one by one, and the result keeps those that are recorded. The key's names
need not be the source's: a synonym is counted under the name the source
accepts for it, and the result keeps the key's own label. Results at any
rank work: a genus or a family counts the records of all its species.
The sources are asked one taxon at a time, so a key of 70 taxa takes about 150 requests of GBIF (about half a minute). If a request fails, it is an error that says which taxon was being asked about, and nothing is pruned.
A dimoose key, pruned. Its attribute "records"
(attr(x, "records")) is a data frame with a row for every taxon of the
original key: Taxon, name (the name looked up), matched (the name
the source counted it under), id (the source's id for that),
records (NA when the source does not know the name) and kept. The
attribute "place" says which place was used. When every taxon is kept
the key is returned unpruned. It is an error if the source knows none of
the key's taxa by name, or if fewer than two taxa are left.
Each name is matched to the GBIF Backbone (/v1/species/match) and its
occurrences in the place are counted (/v1/occurrence/search with
limit=0), two requests per taxon. A synonym is counted under its
accepted name, and a species that GBIF files under one of its subspecies
is counted as the species. A name GBIF has in more than one kingdom is
settled by kingdom, or, for a genus, by taking the accepted genus.
Only records with occurrenceStatus=PRESENT are counted. place can be
a GADM id, such as "USA.27.15_1" for Flathead
County, Montana, or "USA.27_1" for Montana (gadmGid);
a two-letter country code, such as "CH" (country);
a polygon in WKT, "POLYGON((...))", with its points anticlockwise
(geometry);
a named list of GBIF occurrence search parameters, used as they are,
such as list(country = "US", stateProvince = "Montana");
the name of a region, looked up in GADM (/v1/geocode/gadm/search).
Add the regions it lies in after commas to tell places of the same name
apart: "Flathead County, Montana", "Flathead County, MT",
"Montana, USA". A name that still fits several regions, or none, is
an error that lists the regions GADM has with their ids; give the id of
the one you mean. The message printed at the end names the region used.
The place is found with /v1/places/autocomplete (or give its numeric
id). Parts after commas pick among places of one name
("Lake County, MT"). When several places match, the one of exactly that
name is used (if there are several, a country or state before a county
or a park, then the largest), and a message names it and lists the others
with their ids, so that you can give the id of another. The species
observed in the place are listed with /v1/observations/species_counts,
500 a page, counting research-grade observations only. A result of the key
is counted by name when a listed species has that name; otherwise it is
looked up with /v1/taxa (which also finds synonyms) and counts every
listed species below it, or, for a subspecies or variety, its own
observations. Give kingdom to list only that group's species, which
saves pages. When the list would take more requests than asking about
each taxon (a whole country, with no kingdom), a message says so and
each taxon's observations in the place are counted with two requests
instead. Requests made by the default fetch are a second apart, as
iNaturalist asks.
flathead <- regionalKey(montanaKey(), "USA.27.15_1", kingdom = "Plantae") flathead attr(flathead, "records") regionalKey(montanaKey(), "Flathead County", source = "inat", kingdom = "Plantae")flathead <- regionalKey(montanaKey(), "USA.27.15_1", kingdom = "Plantae") flathead attr(flathead, "records") regionalKey(montanaKey(), "Flathead County", source = "inat", kingdom = "Plantae")
Sniglets start with coined names ("dulmzil") because nobody has looked at
them yet. After looking at what a sniglet picks out (the example crops in
the wizard's glossary), give it a real name. The coined word stays as the
sniglet's permanent identifier, so scores, machine tests and saved name
tables keep working; only the text people read changes: the leads
("Has round headlights"), the questions and the glossary.
renameSniglets(x, names = NULL, definitions = NULL)renameSniglets(x, names = NULL, definitions = NULL)
x |
A key from |
names |
The new names: a named character vector
( |
definitions |
Optional named character vector of definitions, named
like |
The names table has one row per sniglet you want to rename, with columns
sniglet (the coined word, or the sniglet's current name), name and
optionally definition. Rules:
a blank name puts the coined word back;
no two sniglets may share a name, and a name may not be another sniglet's coined word;
a definition column replaces the definitions of the rows listed
(blank removes one); without the column, definitions are left alone;
a used column, as snigletNames() returns, is ignored.
Names read best as noun phrases ("an egg-crate grille"), since leads are
written "Has name" and "Lacks name".
A renamed copy of x; x itself is not changed.
set.seed(1) patches <- array(rnorm(5 * 9 * 8), c(5, 9, 8)) patches <- sweep(patches, c(1, 2), sqrt(apply(patches^2, c(1, 2), sum)), "/") s <- discoverSniglets(patches, k = 3) first <- s$sniglets$sniglet[1] s <- renameSniglets(s, stats::setNames("round headlights", first), definitions = stats::setNames("Circular lamps in the grille", first)) snigletNames(s)set.seed(1) patches <- array(rnorm(5 * 9 * 8), c(5, 9, 8)) patches <- sweep(patches, c(1, 2), sqrt(apply(patches^2, c(1, 2), sum)), "/") s <- discoverSniglets(patches, k = 3) first <- s$sniglets$sniglet[1] s <- renameSniglets(s, stats::setNames("round headlights", first), definitions = stats::setNames("Circular lamps in the grille", first)) snigletNames(s)
Embedding images needs a vision model, and so Python; using embeddings
does not. saveEmbeddings() writes what embedImages() or embedTexts()
returned to one small file, and loadEmbeddings() reads it back as it
was, row names and tiling included, on a computer with no Python at all.
Everything dimoose does after embedding (discoverSniglets(),
snigletScores(), keyFromSniglets(), keyFromClusters(),
featureScores(), ...) works from the loaded embeddings.
saveEmbeddings(x, file, precision = c("float32", "float16"), model = NULL) loadEmbeddings(file)saveEmbeddings(x, file, precision = c("float32", "float16"), model = NULL) loadEmbeddings(file)
x |
A numeric matrix or array, or a list of them such as
|
file |
File to write or read; |
precision |
|
model |
Optional |
saveEmbeddings() returns file, invisibly. loadEmbeddings()
returns what was saved: a matrix or array, or a list of them. If the
file records a model, its details are in the result's model attribute.
The file is in the
safetensors format, which
Python (safetensors.numpy.load_file()), JavaScript and other languages
read too:
8 bytes: the length N of the header in bytes, an unsigned 64-bit
little-endian integer;
N bytes: the header, one JSON object in UTF-8, padded with spaces so
that N is a multiple of 8. It has one entry per array, in the order
the arrays are stored, named image and patches (for
embedImages()), embeddings (for a single matrix) or by the names of
the list given. Each entry has dtype ("F32" or "F16"), shape
(the dimensions, e.g. [36, 512]) and data_offsets ([begin, end]:
the array's first byte and the byte after its last, counted from the end
of the header);
the arrays' numbers, one array after another with nothing between them,
in row-major (C) order: an images x d matrix is stored image by image,
an images x tiles x d array image by image and tile by tile.
Numbers are IEEE 754 floats of 4 bytes ("F32") or 2 bytes ("F16"), and
always little-endian, whatever the computer that writes or reads the file.
The header's __metadata__ entry is an object of strings: format
("dimoose-embeddings"), version ("1"), single ("true" when x
was one array and not a list, so that it is read back as one),
rownames.<array> (a JSON array of the row names: image ids or texts;
other dimensions' names are not kept), tiles.<array> and
empty.<array> (the tiling and the rule for empty tiles, see
embedImages(), as numbers separated by commas) and model (a JSON
object with the model's details, if there is a model).
loadEmbeddings() also reads safetensors files written by other programs,
as long as their arrays are 16-, 32- or 64-bit floats.
R keeps numbers as 64-bit floats and the file holds 32-bit ones (about 7
significant digits). Vision models compute in 32 bits, so what
embedImages() and embedTexts() return is saved and read back exactly;
other numbers come back rounded to the nearest 32-bit float. With
precision = "float16" every number is rounded to about 3 significant
digits: unit-length embeddings change by at most 0.0005 per coordinate.
Numbers too large for the chosen precision become Inf, and NA is read
back as NaN.
The embeddings of the package's example photos are in its
extdata/embeddings folder, so the examples can be followed with no
Python: pics-clip-vit-b-32.safetensors (the 36 fruit photos in
extdata/pics, whole photos and 4 x 4 tiles),
new-photos-clip-vit-b-32.safetensors (the 6 photos in
extdata/new-photos), texts-clip-vit-b-32.safetensors (the prompts of
the vision vignette) and montana-bioclip.safetensors (the tree photos in
extdata/montana and the names of montanaKey()'s species, embedded with
BioCLIP). The folder's README.md says how each was made.
# stand-ins for embedImages(model, images): 3 images, 2 tiles each, 4 dimensions emb <- list(image = matrix(rnorm(12), 3, 4, dimnames = list(c("a", "b", "c"), NULL)), patches = array(rnorm(24), c(3, 2, 4), dimnames = list(c("a", "b", "c"), NULL, NULL))) attr(emb$patches, "tiles") <- 1L file <- tempfile(fileext = ".safetensors") saveEmbeddings(emb, file) back <- loadEmbeddings(file) all.equal(back$image, emb$image, tolerance = 1e-6) rownames(back$image) attr(back$patches, "tiles") # the example photos' embeddings, made with CLIP ViT-B/32: a key with no Python pics <- loadEmbeddings(system.file("extdata", "embeddings", "pics-clip-vit-b-32.safetensors", package = "dimoose")) dim(pics$patches) attr(pics, "model")$arch images <- imageSet(system.file("extdata", "pics", package = "dimoose")) sniglets <- discoverSniglets(pics$patches, k = 40) scores <- snigletScores(pics$patches, sniglets$centroids) keyFromSniglets(scores, images, sniglets, method = "rpart")# stand-ins for embedImages(model, images): 3 images, 2 tiles each, 4 dimensions emb <- list(image = matrix(rnorm(12), 3, 4, dimnames = list(c("a", "b", "c"), NULL)), patches = array(rnorm(24), c(3, 2, 4), dimnames = list(c("a", "b", "c"), NULL, NULL))) attr(emb$patches, "tiles") <- 1L file <- tempfile(fileext = ".safetensors") saveEmbeddings(emb, file) back <- loadEmbeddings(file) all.equal(back$image, emb$image, tolerance = 1e-6) rownames(back$image) attr(back$patches, "tiles") # the example photos' embeddings, made with CLIP ViT-B/32: a key with no Python pics <- loadEmbeddings(system.file("extdata", "embeddings", "pics-clip-vit-b-32.safetensors", package = "dimoose")) dim(pics$patches) attr(pics, "model")$arch images <- imageSet(system.file("extdata", "pics", package = "dimoose")) sniglets <- discoverSniglets(pics$patches, k = 40) scores <- snigletScores(pics$patches, sniglets$centroids) keyFromSniglets(scores, images, sniglets, method = "rpart")
Embeds the images (once per distinct crop region the key's text features
use) and applies featureScores().
scoreImages(key, model, paths)scoreImages(key, model, paths)
key |
A key with a |
model |
|
paths |
Image files, or an |
An images x features matrix for classify().
Lists the sniglets of a key (or of a discoverSniglets() result) with the
names they currently go by. This is the table renameSniglets() takes:
write it to a CSV file, fill in the name and definition columns, and
read it back.
snigletNames(x)snigletNames(x)
x |
A key from |
A data frame with one row per sniglet:
sniglet: the coined word. It identifies the sniglet and never changes.
name: the name people see. It is the coined word until you rename it.
definition: an optional sentence saying what the sniglet is.
used (keys only): whether the key asks about this sniglet. Only these
need names.
Score images against sniglets
snigletScores(patches, centroids) termScores(patches, centroids)snigletScores(patches, centroids) termScores(patches, centroids)
patches |
As in |
centroids |
A |
An images x sniglets matrix: for each sniglet, the highest cosine
similarity between the centroid and any patch of the image. Columns are
named by the coined words, whatever the sniglets have been renamed to.
Photos of one observation show the same individual, often seconds apart. Putting one in the training set and another in the test set would make a key look better than it is, so whole observations go to one side.
splitByObservation(observation, label, test = 0.3, seed = 1)splitByObservation(observation, label, test = 0.3, seed = 1)
observation |
The observation each photo belongs to. A photo without
one ( |
label |
The class of each photo. The split is made within each class, so every class keeps about the same share for testing. |
test |
Share of each class's observations to test with. |
seed |
Seed for the draw. |
A character vector, "train" or "test" for each photo. A class
with two or more observations gets at least one on each side; a class
with a single observation goes to the training set. An observation with
photos in several classes stays whole too: all its photos go to the
training set, so nothing in the test set has a sibling in training.
obs <- c(1, 1, 2, 3, 4, 5, 5, 6) label <- rep(c("oak", "elm"), each = 4) data.frame(obs, label, split = splitByObservation(obs, label, test = 0.5))obs <- c(1, 1, 2, 3, 4, 5, 5, 6) label <- rep(c("oak", "elm"), each = 4) data.frame(obs, label, split = splitByObservation(obs, label, test = 0.5))
A sniglet starts with a coined word ("dulmzil") because nobody has looked
at it yet. This function does the first look for you: it shows a
vision-language model each sniglet's example crops (and, for contrast,
crops from photos in which the sniglet was not found, when there are any)
and asks for a short name and a one-sentence definition. The result is a
names table for renameSniglets(), so naming becomes reviewing suggestions
instead of writing every name from scratch. Nothing is renamed until you
pass the table (or the CSV file) to renameSniglets().
suggestSnigletNames( x, model, vision = NULL, images = NULL, scores = NULL, crops = NULL, sniglets = NULL, labels = NULL, examples = 4, contrast = 4, maxWords = 3, retries = 2, file = NULL, pause = 0, size = 96, resume = FALSE )suggestSnigletNames( x, model, vision = NULL, images = NULL, scores = NULL, crops = NULL, sniglets = NULL, labels = NULL, examples = 4, contrast = 4, maxWords = 3, retries = 2, file = NULL, pause = 0, size = 96, resume = FALSE )
x |
A key from |
model |
The vision-language model: a function |
vision, images
|
A |
scores |
Optional |
crops |
Optional crops of your own: a list named by coined word, each
entry either a character vector of examples or a list with |
sniglets |
Which sniglets to name (coined words or current names). By default the ones a key uses, or all of a sniglet set. |
labels |
The class names, which a sniglet name must not contain. By
default the key's results, or the labels of |
examples, contrast
|
How many example and contrast crops to show per sniglet at most. |
maxWords |
Longest name allowed, in words. |
retries |
How many times to ask again when a name cannot be used. |
file |
Optional path of a CSV file to write the table to. It is the
spreadsheet |
pause |
Seconds to wait between sniglets, for APIs with a rate limit. |
size |
Size in pixels of crops cut with |
resume |
|
A data frame with one row per sniglet, in the shape
renameSniglets() takes: sniglet (the coined word), name and
definition (the suggestion, or what the sniglet already had when there
is no usable suggestion), followed by status ("ok", "suffixed",
"rejected", "error" or "no crops") and note. renameSniglets()
ignores the last two columns.
A key that has been through patchExemplars() carries its crops, so
nothing else is needed. The contrast crops are those of the key's "Lacks"
leads.
For a key without crops, or a discoverSniglets() result, give vision
and images and the crops are cut for you. For a sniglet set, scores
(from snigletScores()) adds contrast crops: the best-matching region of
the photos that match the sniglet least.
Or give your own in crops.
Each suggestion is checked, and the model is asked again (up to retries
times, with the reason) when a name
has more than maxWords words (a leading "a", "an" or "the" is not
counted);
contains the name of one of the classes (labels), also in the plural:
a sniglet is something you can see on a specimen, such as "a serrate
margin", not the species it was found on. For labels written
"Common name (Genus species)" the common name, the scientific name and
the genus are all refused, and so is the last word of a common name of
several words ("aspen" of "Quaking aspen", so "aspen bark" is refused),
unless every class ends in it or it is a general word such as "tree" or
"flower";
is already the name or the coined word of another sniglet. Capitals, punctuation and a leading article make no difference: "Serrate margin" and "a serrate margin" are the same name;
is no name at all ("unknown", "n/a", or nothing but punctuation).
A leading "has" or "lacks" is taken off first, since the key adds its own.
If the last answer is still a duplicate it gets a number ("peeling bark 2", status "suffixed"). If it still breaks another rule, or the model
fails (an API error, a rate limit, a refusal, a reply that cannot be
read), the sniglet keeps the name it had and the row says why in status
and note; the other sniglets are not affected.
With file, the table is written after every sniglet, so a run that is
interrupted or stopped by an error leaves what it had. resume = TRUE
picks such a file up: sniglets it already has a usable name for (status
"ok" or "suffixed") are not asked about again, the others are. Without
resume an existing file is overwritten. The folder of file must exist.
claudeNamer(), renameSniglets(), snigletNames()
set.seed(1) patches <- array(rnorm(5 * 9 * 8), c(5, 9, 8)) patches <- sweep(patches, c(1, 2), sqrt(apply(patches^2, c(1, 2), sum)), "/") s <- discoverSniglets(patches, k = 2) pics <- imageSet(system.file("extdata", "pics", package = "dimoose")) crops <- stats::setNames(list(pics$path[1:2], pics$path[3:4]), s$sniglets$sniglet) # a stand-in for a real model, which would look at the crops replies <- c('{"name": "a waxy skin", "definition": "Smooth skin with a dull shine."}', '{"name": "a ribbed surface", "definition": "Parallel ridges along the fruit."}') asked <- 0 fake <- function(images, prompt) { asked <<- asked + 1; replies[asked] } suggestions <- suggestSnigletNames(s, fake, crops = crops, labels = unique(pics$label)) suggestions snigletNames(renameSniglets(s, suggestions))set.seed(1) patches <- array(rnorm(5 * 9 * 8), c(5, 9, 8)) patches <- sweep(patches, c(1, 2), sqrt(apply(patches^2, c(1, 2), sum)), "/") s <- discoverSniglets(patches, k = 2) pics <- imageSet(system.file("extdata", "pics", package = "dimoose")) crops <- stats::setNames(list(pics$path[1:2], pics$path[3:4]), s$sniglets$sniglet) # a stand-in for a real model, which would look at the crops replies <- c('{"name": "a waxy skin", "definition": "Smooth skin with a dull shine."}', '{"name": "a ribbed surface", "definition": "Parallel ridges along the fruit."}') asked <- 0 fake <- function(images, prompt) { asked <<- asked + 1; replies[asked] } suggestions <- suggestSnigletNames(s, fake, crops = crops, labels = unique(pics$label)) suggestions snigletNames(renameSniglets(s, suggestions))
Compares each photo with the names of the key's results and ranks the
results by how well they match, so a photo can be checked against the
species the key covers. This works best with a model trained on names of
organisms, such as BioCLIP 2 (visionModel(name = "bioclip-2")): it
recognises a species from its scientific name without being shown any
examples. Only the key's own species are considered, which makes the
answers much more reliable than asking the model to name a species out of
all those it knows.
suggestSpecies(key, model, paths, top = 3, prompt = "a photo of %s.")suggestSpecies(key, model, paths, top = 3, prompt = "a photo of %s.")
key |
A dimoose key. |
model |
|
paths |
Image files, or an |
top |
How many of the best-matching results to return per photo. |
prompt |
The text each name is put into, with |
Each result is described by its scientific name when it is written as
"Common name (Genus species)", and by the whole result otherwise. The
probabilities are a softmax over all the key's results with the
temperature CLIP models are trained with (cosine similarity times 100).
A data frame with top rows per image: image (its id), rank,
Taxon, probability and similarity (cosine similarity).
identifyPhotos() for one answer per photo, at the genus or
family when the species is uncertain.
## Not run: model <- visionModel(name = "bioclip-2") suggestSpecies(montanaKey(), model, "my-photo.jpg") ## End(Not run)## Not run: model <- visionModel(name = "bioclip-2") suggestSpecies(montanaKey(), model, "my-photo.jpg") ## End(Not run)
Writes the key's spanning tree as an Auspice v2 dataset, which the
Nextstrain viewer and auspice.us can display. Node names are made unique
and space-free (required by the schema); the original labels and each
lead's text are kept in node_attrs. div is the depth from the root.
toAuspiceJSON(key, title = NULL, file = NULL)toAuspiceJSON(key, title = NULL, file = NULL)
key |
A dimoose key. |
title |
Dataset title; defaults to the key's description. |
file |
Optional path to write to; if |
The JSON string (invisibly if written to file).
js <- toAuspiceJSON(phylotreeKey("H2a"))js <- toAuspiceJSON(phylotreeKey("H2a"))
Unlike toNewick() and toDataTree(), which use a spanning tree, this
keeps the key's true structure: a couplet reached from several leads has
several incoming edges. Each edge carries the lead's label and
Character text; vertices are couplets and taxa.
toIgraph(key)toIgraph(key)
key |
A dimoose key. |
An igraph::igraph directed graph.
## Not run: g <- toIgraph(arachnidaKey()) igraph::vcount(g) ## End(Not run)## Not run: g <- toIgraph(arachnidaKey()) igraph::vcount(g) ## End(Not run)
The dimoose method $toNewick() writes quoted Newick, which is correct but
which several parsers (ape, scikit-bio, toytree) mangle. This wrapper adds
two options for interoperating with the wider phylogenetics ecosystem.
toNewick( key, labels = c("quoted", "safe"), annotate = c("none", "nhx", "beast") )toNewick( key, labels = c("quoted", "safe"), annotate = c("none", "nhx", "beast") )
key |
A dimoose key. |
labels |
|
annotate |
[ ]: R:%20 |
For labels = "quoted" or any annotate, a single Newick string.
For labels = "safe" with annotate = "none", a list of newick (the
string) and labels (a data frame of id and label).
k <- arachnidaKey() cat(toNewick(k, annotate = "nhx"))k <- arachnidaKey() cat(toNewick(k, annotate = "nhx"))
Builds a treedata object whose tree is the key's spanning tree and whose
node data carries each lead's Character, Choice and Taxon. The data
of a child node describes the edge into it, so in ggtree you draw lead text
on branches with geom_label(aes(x = branch, label = Character)) and join
your own data by the node column (never by label, since taxa repeat).
toTreedata(key)toTreedata(key)
key |
A dimoose key. |
A tidytree::treedata object.
## Not run: td <- toTreedata(arachnidaKey()) library(ggtree) ggtree(td) + geom_tiplab() + geom_label(aes(x = branch, label = Character)) ## End(Not run)## Not run: td <- toTreedata(arachnidaKey()) library(ggtree) ggtree(td) + geom_tiplab() + geom_label(aes(x = branch, label = Character)) ## End(Not run)
Biochemical and growth test results for 74 Vibrio taxa from Noguerola &
Blanch (2008), one row per taxon. vibrioKey() generates a key from it
and vibrioMatrix() recodes it for keyFromMatrix().
vibriovibrio
A data frame with 74 rows and 46 columns: species, then 45 tests
coded as in the paper: + positive, - negative, (+) / (-) mostly
positive / negative, v or d variable, ND not determined (and one
blank cell in the source, NA). Tests under the paper's "Acid from" and
"Resistance to" headings are named with that prefix.
Noguerola I, Blanch AR (2008). Identification of Vibrio spp.
with a set of dichotomous keys. Journal of Applied Microbiology 105,
175-185. doi:10.1111/j.1365-2672.2008.03730.x. Extracted from the
paper's table; see data-raw/vibrio/ in the package sources.
Recodes the vibrio consensus matrix of Noguerola & Blanch (2008) into the
character matrix keyFromMatrix() takes: + and (+) (mostly positive)
become "+", - and (-) become "-", the variable results v and d
become "+/-", and ND (not determined) and the one blank cell become
NA.
vibrioMatrix()vibrioMatrix()
A data frame with 74 rows: Taxon, then the 45 tests.
m <- vibrioMatrix() m[1:4, 1:4] table(unlist(m[-1]), useNA = "ifany")m <- vibrioMatrix() m[1:4, 1:4] table(unlist(m[-1]), useNA = "ifany")
Uses Python's open_clip through reticulate. You do not need to install
anything in Python yourself: the first call sets up a private Python with
the packages dimoose needs (torch, open_clip_torch, pillow, numpy)
and reuses it afterwards. That first call downloads them, which takes a
few minutes. This needs reticulate 1.41 or later.
visionModel( weights = NULL, name = "ViT-B-32", pretrained = "openai", python = NULL, backend = c("torch", "onnx"), precision = c("fp32", "fp16", "int8") )visionModel( weights = NULL, name = "ViT-B-32", pretrained = "openai", python = NULL, backend = c("torch", "onnx"), precision = c("fp32", "fp16", "int8") )
weights |
Local checkpoint file: OpenAI's |
name |
The model: an open_clip architecture name, |
pretrained |
open_clip's tag for the weights to download for an
architecture |
python |
Optional path to the Python executable to use (passed to
|
backend |
|
precision |
For |
An object of class dimooseVisionModel with py (the Python model)
and spec (library, architecture, weights SHA-256, image size, ...).
Most of that first download is PyTorch. visionModel(backend = "onnx")
runs the same model on ONNX Runtime instead, which needs only the Python
packages onnxruntime, tokenizers, pillow and numpy (about 150 MB
installed) and gives the same embeddings (cosine 1.0000 with the PyTorch
ones on dimoose's example photos). CLIP ViT-B/32 and ViT-B/16 are
downloaded ready-made; BioCLIP and other models are converted once with
exportOnnxModel(). See onnxModel(), which this calls, for where models
come from and for precision. Use one backend per R session: the Python
that reticulate sets up for the first one lacks the other's packages.
To use a Python of your own instead, give its path in python, or select
it with reticulate::use_python(), reticulate::use_virtualenv() or the
RETICULATE_PYTHON environment variable before the first call, and install
the packages there with
reticulate::py_install(c("torch", "open_clip_torch", "pillow", "numpy")).
Any open_clip model whose image side is a vision transformer works. The key builders only see its embeddings, so nothing else changes.
visionModel(name = "bioclip") loads
BioCLIP and
visionModel(name = "bioclip-2") BioCLIP 2, models trained on images of
organisms, from the Hugging Face Hub. They are downloaded on first use.
name = "hf-hub:<organisation>/<model>" loads any other open_clip model
on the Hub, and name = "local-dir:<folder>" a copy you have downloaded
(the folder holds the model's open_clip_config.json and weights file).
name and pretrained together select one of open_clip's built-in
models, e.g. name = "ViT-B-16", pretrained = "laion2b_s34b_b88k".
A key can only be followed with the model it was built with: use the same
model for embedImages() and later for scoreImages().
BioCLIP's text side was trained on names of organisms, not on descriptions
of what they look like. It suits sniglets (discoverSniglets()), which use
images only, better than vocabulary questions (keyFromClusters()).
## Not run: model <- visionModel() # CLIP ViT-B/32 model <- visionModel(name = "bioclip") # BioCLIP, for photos of organisms model <- visionModel(backend = "onnx") # CLIP ViT-B/32 without PyTorch ## End(Not run)## Not run: model <- visionModel() # CLIP ViT-B/32 model <- visionModel(name = "bioclip") # BioCLIP, for photos of organisms model <- visionModel(backend = "onnx") # CLIP ViT-B/32 without PyTorch ## End(Not run)
Prompts for a vocabulary of visual features
vocabularyPrompts(vocabulary, template = "{x}")vocabularyPrompts(vocabulary, template = "{x}")
vocabulary |
Data frame with |
template |
Text with |
A character vector: the feature prompt then the opposite prompt
for each row, in order. Embed these with embedTexts() and pass the
result to keyFromClusters().
v <- data.frame(feature = c("round lights", "a flat hood"), opposite = c("square lights", "a curved hood")) vocabularyPrompts(v, "a truck with {x}")v <- data.frame(feature = c("round lights", "a flat hood"), opposite = c("square lights", "a curved hood")) vocabularyPrompts(v, "a truck with {x}")