Package 'dimoose'

Title: Tools to Reshape and Manipulate Dichotomous Keys Used in Taxonomic Identification
Description: Transforms between dichotomous identification keys and a variety of phenotype formats, including feature matrices, and generates interactive dichotomous key wizards. Imports keys from 'FishBase' <https://www.fishbase.se/keys/allkeys.php> and includes PhyloTree Build 17, the human mitochondrial DNA phylogeny (van Oven 2015, <doi:10.1016/j.fsigss.2015.09.155>), as distributed by 'Haplogrep' 3.
Authors: Jeremy Leipzig [aut, cre] (ORCID: <https://orcid.org/0000-0001-7224-9620>)
Maintainer: Jeremy Leipzig <[email protected]>
License: GPL (>= 3)
Version: 0.3.0
Built: 2026-11-21 04:17:38 UTC
Source: https://github.com/leipzig/dimoose

Help Index


Score aligned sequences against a key built from an alignment

Description

Computes the tests of a keyFromAlignment() key for each sequence, for classify(). The sequences must be aligned to the same coordinates the key was built with: the same alignment columns or, for a key built with a reference, an alignment that contains that reference sequence.

Usage

alignmentScores(key, alignment, reference = NULL)

Arguments

key

A key from keyFromAlignment().

alignment

As in keyFromAlignment().

reference

Sequence to number sites by; by default the reference the key was built with, if any.

Value

A ⁠sequences x features⁠ matrix: 1 if the sequence has the state the first lead of a couplet asks for, 0 if it has another state, and NA if it has a gap or an ambiguity code there.


Key to the Orders of Arachnida

Description

The key from Triplehorn & Johnson (2005), Borror and DeLong's Introduction to the Study of Insects, 7th ed., chapter 5, as a table of leads (see keyFromLeads()); arachnidaKey() turns it into a dimoose key. Transcribed by hand from the book.

Usage

arachnida

Format

A data frame with 22 rows (11 couplets) and 6 columns:

Statement

Couplet number.

Choice

a for the first lead (1.), b for the second (⁠1'.⁠).

Character

Text of the lead.

Next

Couplet the lead leads to, or "-".

Taxon

Order the lead ends in, for terminal leads.

Page

Page of the book where that order is treated.

Source

Triplehorn CA, Johnson NF (2005). Borror and DeLong's Introduction to the Study of Insects, 7th ed. Thomson Brooks/Cole. Transcription: ⁠data-raw/arachnida/⁠ in the package sources.


Benchmark identification on field photos

Description

Measures how often photos are identified correctly, by one of three routes:

Usage

benchmarkKey(
  x,
  photos,
  model = NULL,
  route = NULL,
  top = 3,
  test = NULL,
  split = NULL,
  seed = 1,
  dir = NULL,
  prompt = "a photo of %s.",
  lineage = NULL,
  confidence = 0.8,
  k = 40,
  tiles = 4,
  method = "rpart",
  embeddings = NULL,
  reps = 1
)

Arguments

x

A dimoose key, or (for route = "build") a visionModel().

photos

The photos: the manifest inatPhotos() returns, the path of a manifest.csv (give dir too), or a data frame with path and label and optionally id and observation. Labels may be the key's results or just their scientific names. Without an observation column every photo counts as its own observation.

model

A visionModel().

route

"names", "key" or "build". By default "build" when x is a vision model, "key" for a key with machine tests and "names" for any other key.

top

Report the accuracy within this many guesses ("names" only: a key gives one answer).

test

Share of each species' observations to test with. By default 0.3 for "build" and all of them for the other routes, which learn nothing from the photos.

split

Instead of test: "train" or "test" for each photo, e.g. from splitByObservation(), to compare routes on the same test photos.

seed

Seed for the split and for finding sniglets.

dir

The folder the manifest's file column is relative to.

prompt, lineage, confidence

As in identifyPhotos(). lineage is optional and is never looked up here.

k, tiles, method

For "build": number of sniglets (discoverSniglets()), tiling (embedImages()) and tree method (keyFromSniglets()).

embeddings

For "build": the result of embedImages() for photos, if you have it already (rows in the same order).

reps

Number of splits to run. One split of a small sample says little: six test photos per species move a species' accuracy by 17 points each. With reps above 1 the benchmark is repeated with seeds seed, seed + 1, ..., each with a new split (and, for "build", a new key; the photos are embedded once), and the spread is reported. Needs a test share below 1, and no split.

Details

  • "names": the model ranks the key's own species for each photo from their names alone (suggestSpecies()); nothing is learned from the photos. With a lineage the answers of identifyPhotos(), which fall back to the genus, family or order when the species is uncertain, are scored too.

  • "key": a key with machine tests (from keyFromSniglets() or keyFromClusters()) is followed for each photo (scoreImages(), evaluateKey()).

  • "build": a photo key is built from the training photos (sniglets found in them, then keyFromSniglets()) and followed for the test photos. This is the route to use for "how good a key can be learned from photos like these".

Training and test photos are divided by observation with splitByObservation(), never photo by photo.

Value

An object of class dimooseBenchmark, a list with

  • route, n (test photos), nTrain, observations (test observations);

  • accuracy: share of test photos identified correctly (top-1);

  • interval: its 95% Wilson interval (lower, upper), which treats the test photos as independent draws: read it as the least uncertainty there is, not all of it;

  • topK: named vector of the accuracy within 1, ..., top guesses;

  • perSpecies: for each species its number of test photos, top1 and (for "names") the accuracy within top guesses. Every species of the photos (and of the key) has a row; one without test photos has n = 0 and NA accuracies;

  • confusion: table of actual (rows) against predicted (columns), with the same species in the same order on both sides, so that the row sums are the n of perSpecies and the diagonal holds the correct answers. Any other answer has a column after them: a leaf that names several species, and "<none>" when a key gave no answer;

  • results: one row per test photo with id, observation, actual, predicted, rank (where the true species came in the ranking) and correct;

  • split: "train" or "test" for every photo given;

  • with reps above 1, all of the above describe the first split, and there are also repeats (one row per split: rep, seed, n, nTrain, correct, accuracy), accuracyMean, accuracySd (the standard deviation between splits) and accuracyRange, and perSpecies gains top1Mean, the species' accuracy over all splits' test photos together. The splits share photos, so they are not independent samples: the spread shows how much one split can mislead, it is not a standard error;

  • key (for "build", the key that was built) and identify (for "names" with a lineage: the identifyPhotos() answers with correct, and the shares that were confident and correct).

See Also

inatPhotos(), evaluateKey(), looKey()

Examples

## Not run: 
key <- montanaKey()
photos <- inatPhotos(key, n = 20, dir = file.path(tempdir(), "inat"))
model <- visionModel(name = "bioclip-2")
benchmarkKey(key, photos, model)                    # species ranked from their names
benchmarkKey(model, photos)                         # a key built from 70% of the observations

## End(Not run)

Check a key's species names against GBIF

Description

Looks up the scientific name of every result in the key in the GBIF Backbone Taxonomy and reports names that are misspelled, that are synonyms of a name GBIF now accepts, or that GBIF does not know. A result written as "Common name (Genus species)" is checked by the name in parentheses; other results are checked when they look like a scientific name.

Usage

checkNames(key, kingdom = NULL)

Arguments

key

A dimoose key.

kingdom

Optional kingdom ("Plantae", "Animalia", "Fungi", ...), which helps GBIF tell apart names that are used in more than one kingdom.

Details

This needs an internet connection: one request to GBIF per name, and one more for each synonym.

Value

A data frame with one row per name checked: Taxon (as in the key), name (the scientific name checked), status ("ok", "misspelled", "synonym", "not found", or "unchecked" when GBIF could not be reached), gbif (the name GBIF matched), accepted (the name GBIF accepts for it), genus, family and order (from GBIF's classification, used by identifyPhotos()) and note (what to fix, in words).

See Also

keyFromSpreadsheet(checkNames = TRUE), which runs this check and reports problems by spreadsheet row.

Examples

checkNames(montanaKey(), kingdom = "Plantae")

Follow a key by machine

Description

Walks each row of scores through key: at every couplet the lead whose test (Feature, Test, Threshold) passes is taken. When no lead or more than one passes (for instance on a missing score), the result is NA.

Usage

classify(key, scores)

Arguments

key

A dimoose key with Feature, Test and Threshold lead columns, e.g. from keyFromTerms().

scores

A numeric matrix, one row per object, with a column for every feature id the key tests (see featureScores() and scoreImages()).

Value

A data frame with id (row names of scores), result and path (the leads taken, e.g. "1a 3b").


A sniglet namer that asks Claude

Description

Makes the model function suggestSnigletNames() needs from the Anthropic Messages API: each call sends one sniglet's crops and the prompt, and returns the text of the reply. Every call is billed to the API key's account; one call is made per sniglet, plus one for every name that has to be asked for again.

Usage

claudeNamer(
  model = "claude-opus-5-5",
  apiKey = Sys.getenv("ANTHROPIC_API_KEY"),
  maxTokens = 16000,
  effort = NULL,
  retries = 3,
  timeout = 120,
  url = "https://api.anthropic.com/v1/messages",
  fallback = FALSE
)

Arguments

model

The model to ask. It has to accept images.

apiKey

The API key. By default the ANTHROPIC_API_KEY environment variable.

maxTokens

The most tokens a reply may use. Current models think before they answer and the thinking counts, so leave room for it; only the tokens actually used are billed.

effort

Optional effort level ("low", "medium", "high", ...) for models that take one; "low" is plenty for naming and costs less. By default it is not sent and the model's own default applies.

retries

How many times to try again after a rate limit or a server error.

timeout

Seconds to wait for a reply.

url

The API endpoint.

fallback

TRUE to let the API answer with another model when the one asked declines (its safety classifiers can misjudge a harmless photo). This sends fallbacks = "default" with the beta header server-side-fallback-2026-07-01, which the Anthropic API takes but not every platform that serves Claude; by default it is not sent and a declined request is reported on that sniglet's row.

Details

A rate limit (HTTP 429), an overloaded service (529) or a server or connection error is tried again up to retries times, waiting as long as the API asks. Any other error stops that one call with the API's message, which suggestSnigletNames() reports for that sniglet before carrying on with the next. So does a reply the model declined to give (stop reason "refusal") and one that ran out of tokens ("max_tokens").

Value

A function ⁠function(images, prompt)⁠ returning the reply's text.

Images

The API takes JPEG, PNG, GIF and WebP images of at most 5 MB each (as base64; some platforms allow more), and reads small images poorly. With the magick package installed, crops larger than 2000 pixels or 5 MB are scaled down before they are sent and crops smaller than 200 pixels are enlarged. Without it they are sent as they are, and a crop over 5 MB stops that call with a message. A call with more than 100 images is refused, and with many large crops each is scaled down further so that the request stays under the API's 32 MB.

See Also

suggestSnigletNames()

Examples

## Not run: 
# needs an API key in ANTHROPIC_API_KEY; the calls are billed to it
key <- patchExemplars(vision, key, images)
suggestSnigletNames(key, claudeNamer(), file = "names.csv")
# ... review names.csv ...
key <- renameSniglets(key, "names.csv")

## End(Not run)

A dichotomous key

Description

An R6 object holding a dichotomous key as a table of the leads on every path from the root to a terminal taxon. Usually created by importFishbase() or parseFishbase(). See https://www.fishbase.se/keys/allkeys.php for FishBase keys.

Active bindings

desc

Key title or description (read-only).

df

Data frame with one row per lead (or per term, if terms were separated) per terminal taxon, with columns Statement, Choice, Character, Taxon, and pSt/pCh for the parent lead.

meta

Key/value data frame of metadata (citation, image URLs, ...).

taxa

Data frame of terminal taxa and their resolved names.

leads

Data frame with one row per lead: Statement, Choice, Character, Next (next couplet, or "-" if terminal) and Taxon, plus optional Prev, Image, ImageLink, TaxonUrl, Question, Feature, Test and Threshold. Derived from df when the object was created without one.

features

Data frame of features a machine can compute, one per row (see featureTable()), or NULL for keys meant only for people.

Methods

Public methods


dimoose$new()

Create a new dimoose object.

Usage
dimoose$new(df, desc = NA, meta = NA, taxa = NA, leads = NULL, features = NULL)
Arguments
df

Data frame of leads (see the df field).

desc

Key title or description.

meta

Key/value metadata data frame.

taxa

Data frame of terminal taxa.

leads

Optional data frame of leads (see the leads field).

features

Optional data frame of machine-readable features (see featureTable()).


dimoose$print()

Print a short overview of the key.

Usage
dimoose$print(...)
Arguments
...

Unused.


dimoose$summary()

Counts describing the key and how quickly it reaches a result (see keyQuality()).

Usage
dimoose$summary(weights = NULL)
Arguments
weights

Optional named numeric vector of how common each taxon is, as for keyQuality(); by default all taxa count the same.

Returns

A one-row data frame: title, couplets, leads, taxa, depth (longest path in steps), problems (see validate()), meanLeadSteps and meanTaxonSteps (steps to a result, averaged over terminal leads and over taxa), expectedSteps (averaged over taxa by weights), bound (the expected steps of the best key there could be to these taxa, with couplets no larger than this key's largest) and efficiency (bound / expectedSteps, 1 for a key that could not be shorter).


dimoose$validate()

Check the key: leads to missing couplets, terminal leads without a taxon, couplets with fewer than two leads, couplets that cannot be reached from the first one, and (for keys with features) leads whose machine test is missing or inconsistent.

Usage
dimoose$validate()
Returns

A character vector of problems, invisibly if there are none.


dimoose$toDataTree()

Convert the key to a data.tree::Node: couplets are internal nodes and taxa leaves; each child node carries the lead label and the character text of the lead that leads to it. A couplet reached from several leads is attached under the first.

Usage
dimoose$toDataTree()

dimoose$toNewick()

Write the key's topology in Newick format, with couplets as internal node labels and taxa as leaves (labels quoted when needed). A couplet reached from several leads is placed under the first.

Usage
dimoose$toNewick()
Returns

A single string ending in ⁠;⁠.


dimoose$clone()

The objects of this class are cloneable with this method.

Usage
dimoose$clone(deep = FALSE)
Arguments
deep

Whether to make a deep clone.


Discover sniglets by clustering patch embeddings

Description

A sniglet is a visual feature that a model found and that has no name of its own, so dimoose coins one for it ("dulmzil"). This function runs k-means on every patch embedding of every image and keeps the unit length centroids as the definitions of k sniglets. An image "has" a sniglet when one of its patches is close to that centroid (see snigletScores()). Once you have looked at what a sniglet picks out, give it a real name with renameSniglets().

Usage

discoverSniglets(patches, k = 40, seed = 1, nstart = 10, exemplars = 4)

discoverTerms(...)

Arguments

patches

A numeric array ⁠images x regions x dimensions⁠ (unit length rows), e.g. embedImages(...)$patches; dimnames(patches)[[1]] are the image ids. Rows that are all zero are empty regions and are ignored.

k

Number of sniglets.

seed

Seed for k-means and for inventNames().

nstart

Random starts for stats::kmeans().

exemplars

Number of example regions to record per sniglet, each from a different image.

...

Passed to discoverSniglets().

Details

discoverTerms(), termScores() and keyFromTerms() are the earlier names of discoverSniglets(), snigletScores() and keyFromSniglets(), and still work.

Value

An object of class dimooseSniglets: sniglets (data frame with id, sniglet (the coined word), name (what people see; the coined word until renamed) and definition), centroids (⁠k x d⁠, rows named by coined word), exemplars (data frame sniglet, image, patch, score: the best-matching region of each of the best-matching images), best (for every image, the region that matches each sniglet best) and tiles (the tiling of the regions, see embedImages()).

Examples

set.seed(1)
patches <- array(rnorm(5 * 9 * 8), c(5, 9, 8))
norms <- sqrt(apply(patches^2, c(1, 2), sum))
patches <- sweep(patches, c(1, 2), norms, "/")
dimnames(patches) <- list(letters[1:5], NULL, NULL)
discoverSniglets(patches, k = 3)$sniglets

Embed images

Description

Returns one embedding per image and, for discoverSniglets(), embeddings of local regions of each image.

Usage

embedImages(
  model,
  paths,
  region = NULL,
  patches = TRUE,
  tiles = 4,
  minDetail = 8,
  minFill = 0.5
)

Arguments

model

A visionModel().

paths

Image files, or an imageSet() (its path column is embedded and its id column names the rows).

region

Optional crop c(y0, y1, x0, x1) as fractions, applied to every image before embedding.

patches

If TRUE, also return the local embeddings.

tiles

Size of the grid of tiles: 4 cuts 4 x 4 windows that overlap their neighbours by half. Several sizes give several scales (c(2, 4)). 0 uses the model's patch tokens.

minDetail

Tiles with less detail than this (standard deviation of the grey levels, 0 to 255) are treated as empty background and match nothing.

minFill

In a photo with a plain background (one colour around most of its border), tiles in which the subject covers less than this share are treated as empty too, so that features are about the subject and not about its backdrop. It has no effect on photos without a plain background.

Details

By default the regions are tiles: overlapping windows cut out of the image and embedded one by one, as if each were a whole image. A tile's embedding describes only what is inside it, so tiles find parts and surfaces that different images have in common (a stem end, a dimpled peel). Tiles that are mostly plain background are left out. tiles = 0 uses the model's own patch tokens instead, which is faster (one pass per image) and is what fordera does, but in the last layer of a CLIP model each token carries much of the whole image, so the features found are less local.

Value

A list: image (⁠images x d⁠ matrix, rows named by image id: the file name without extension, or the id of an imageSet()) and patches (⁠images x regions x d⁠ array, or NULL), which remembers the tiling in its tiles attribute.


Embed texts

Description

Embed texts

Usage

embedTexts(model, texts)

Arguments

model

A visionModel().

texts

Character vector, without missing values.

Value

A ⁠texts x d⁠ matrix with the texts as row names (no rows for no texts).


Accuracy of a key followed by machine

Description

Accuracy of a key followed by machine

Usage

evaluateKey(key, scores, labels, groups = NULL)

Arguments

key

A dimoose key with Feature, Test and Threshold lead columns, e.g. from keyFromTerms().

scores

A numeric matrix, one row per object, with a column for every feature id the key tests (see featureScores() and scoreImages()).

labels

True label of each row of scores.

groups

Optional named vector mapping labels to coarser groups (e.g. model years to generations) for a second accuracy figure.

Value

A list: results (one row per object with actual, predicted, correct, groupCorrect), accuracy, groupAccuracy and confusion.


Example keys

Description

Ready-made dimoose keys built from the datasets that come with dimoose. All five are ordinary dimoose objects: they share the same fields (leads, df, meta, taxa, desc) and methods (print(), summary(), validate(), toDataTree(), toNewick()) and all work with exportWizard().

Usage

sharkKey()

arachnidaKey()

vibrioKey(variable = c("both", "missing"))

montanaKey()

Arguments

variable

For vibrioKey(): "both" (default) keys a species that is variable for a test out down both leads; "missing" treats variable results as missing. See keyFromMatrix().

Details

  • sharkKey(): FishBase key 1, Key to the families of sharks in the Western Central Pacific (Compagno 1998), from saved FishBase pages in ⁠inst/extdata/fishbase/⁠; 23 couplets, 24 families.

  • arachnidaKey(): the Key to the Orders of Arachnida from Triplehorn & Johnson (2005), the arachnida dataset; 11 couplets, 11 orders (Schizomida is reached from two couplets).

  • vibrioKey(): a key generated from the vibrio consensus matrix of Noguerola & Blanch (2008) with keyFromMatrix() (the matrix itself is vibrioMatrix()). Tests scored + or ⁠(+)⁠ count as positive and - or ⁠(-)⁠ as negative. With variable = "both" (the default) a species that is variable for a test (v, d) is keyed out down both leads of every couplet that uses the test, and so is a species the test was not determined for (ND): 92 couplets, 74 species, 14 of them reached at two or three results. With variable = "missing" those results are treated as missing and the key is an rpart classification tree (keyFromRpart()), which places such a species on one side of the test with surrogate splits: 73 couplets, each species at one result, but a strain showing the species' other result is keyed out wrongly.

  • montanaKey(): the native broadleaf trees of Montana, written for dimoose (the montana dataset); 10 couplets, 11 species, with photos from Wikimedia Commons. It is built with keyFromSpreadsheet() from inst/extdata/montana/montana-key.csv, a template for writing your own.

  • phylotreeKey(): PhyloTree Build 17.

Value

A dimoose object.

References

Compagno LJV (1998). General remarks. In Carpenter KE, Niem VH (eds.), FAO species identification guide for fishery purposes. The living marine resources of the Western Central Pacific, vol. 2, pp. 1196-1207. FAO, Rome. FishBase key 1: https://www.fishbase.se/keys/description.php?keycode=1.

Triplehorn CA, Johnson NF (2005). Borror and DeLong's Introduction to the Study of Insects, 7th ed., chapter 5. Thomson Brooks/Cole. ISBN 0-03-096835-6.

Noguerola I, Blanch AR (2008). Identification of Vibrio spp. with a set of dichotomous keys. Journal of Applied Microbiology 105, 175-185. doi:10.1111/j.1365-2672.2008.03730.x

Examples

sharks <- sharkKey()
sharks
summary(arachnidaKey())
vibrioKey()$validate()
summary(vibrioKey(variable = "missing"))
## Not run: 
exportWizard(sharkKey(), "sharks.html")

## End(Not run)

Write what a web page needs to identify photos with a key's species

Description

Photo identification as suggestSpecies() and identifyPhotos() do it has two halves: the names of the key's results are embedded as texts, and the photo is embedded as an image and compared with them. The text half can be done ahead of time, here; a web page then needs only the image encoder to identify a photo on the visitor's own device, with no server and no upload. This function writes the files such a page loads.

Usage

exportBrowserModel(
  key,
  model,
  dir,
  encoder = NULL,
  prompt = "a photo of %s.",
  textEmb = NULL
)

Arguments

key

A key whose results are the labels.

model

The onnxModel() (or visionModel()) to embed the labels with. It must be the model the image encoder belongs to.

dir

Folder to write to; created if needed.

encoder

The image encoder for the page: NULL copies the image encoder of model (which must then be an onnxModel()); "hub" points the page at that file on the Hugging Face Hub instead of copying it (for a model that came from there); a file path copies that file; a web address (⁠https://...⁠) is written to the manifest as it is.

prompt

The text each name is put into, with ⁠%s⁠ where the name goes, as in suggestSpecies().

textEmb

Optional text embeddings made earlier, a matrix whose row names are the prompts (from embedTexts() or loadEmbeddings()); the labels are then not embedded again.

Value

The path of model.json, invisibly.

Files written to dir

model.json, the manifest (UTF-8 JSON):

  • format: "dimoose-browser-model"; version: 1.

  • key: desc (the key's description) and taxa (the number of labels).

  • model: arch, pretrained, dim (embedding dimensions), library.

  • image_encoder: file (a file name in dir, or null), url (a web address, or null; one of the two is set), bytes, precision ("int8", "fp16", "fp32" or null if unknown), input (the input's name, "pixel_values"), input_type ("float32"), input_shape (⁠[1, 3, S, S]⁠), output (the output to read, "image_embeds"), normalized (false: divide the output by its length).

  • preprocess, how to turn a photo into the input: resize ("shortest": scale so the shorter side is resize_to pixels, keeping the shape), interpolation ("bicubic"), crop ("center": cut the middle image_size x image_size square), rescale (multiply the 0-255 values by this, 1/255), mean and std (per channel: subtract, then divide), layout ("NCHW"), channels ("RGB").

  • text_embeddings: file ("text_embeddings.bin"), dtype ("float32"), byte_order ("little"), shape (⁠[labels, dim]⁠), order ("row-major": label by label), normalized (true), prompt (the template).

  • labels: one object per row of the text embeddings, in order: taxon (the result as the key's leads write it, which is what to look up in the key), name (the name put into the prompt: the scientific name when the result has one) and prompt (the text that was embedded).

  • logit_scale: 100.

text_embeddings.bin: ⁠labels x dim⁠ 32-bit little-endian floats and nothing else, row by row; in JavaScript, ⁠new Float32Array(await (await fetch(url)).arrayBuffer())⁠.

image_encoder.onnx: the image encoder, unless the manifest gives a url instead.

What the page does

Prepare the photo as preprocess says, run the image encoder (for example with onnxruntime-web), divide image_embeds by its length to get v, and take the dot product of v with each row of the text embeddings. The probability of each label is the softmax of logit_scale times those dot products, which is what suggestSpecies() computes.

Size

The image encoder is the download that matters. Use an "int8" encoder for the web: 96 MB for CLIP ViT-B/32 and 90 MB for BioCLIP as exportOnnxModel() writes them, against about 350 MB unquantised, with the same answers on dimoose's example photos (BioCLIP 2, a larger model, is 310 MB even so). Where that is still too much, "mobileclip-s0" at "fp16" is 23 MB, but it is a general model that does not know species names as BioCLIP does. The text embeddings are tiny (2 KB per label), and are best made with the "fp32" text encoder: onnxModel(name, precision = "int8", textPrecision = "fp32").

Tested with onnxruntime-web

The "int8", "fp16" and "fp32" encoders that exportOnnxModel() writes for CLIP ViT-B/32 and BioCLIP were run in onnxruntime-web 1.31.1 on its WebAssembly backend (under Node.js) from folders this function wrote, with the 14 Montana example photos prepared as preprocess says. All three load and run there. Against the same file run in Python the lowest cosine of the embeddings was 0.9996 for "int8" and 1.0000 for "fp16" and "fp32", and the most likely label was the same for every photo. The "int8" files hold 8-bit weights that are turned back into floats when the model loads, so they use no integer arithmetic a runtime might lack. The WebGPU backend was not tested.

Ready-made ⁠_quantized⁠ files from the Hub (encoder = "hub" with an "int8" model) are quantised differently, with integer arithmetic. That of Xenova/clip-vit-base-patch32 had a lowest cosine of 0.85 with the "fp32" embeddings on the same backend and gave another most likely label for 5 of the 14 photos; its "fp16" file had 1.0000. Point a page at the "fp16" file, or copy in an "int8" encoder from exportOnnxModel().

How the photo is resized matters more than the encoder's precision. preprocess$interpolation names the filter the model was trained with (bicubic, with smoothing when a photo is made smaller, as the Python library Pillow does it). With BioCLIP on the Montana photos, resizing with a bilinear filter instead gave embeddings with a cosine of 0.93 to 0.98 (0.97 on average) with the right ones, and nearest-neighbour 0.84 to 0.95.

See Also

onnxModel(), exportOnnxModel(), suggestSpecies().

Examples

## Not run: 
model <- onnxModel("bioclip", precision = "int8", textPrecision = "fp32")
exportBrowserModel(montanaKey(), model, "site/model")
list.files("site/model")   # image_encoder.onnx, model.json, text_embeddings.bin

## End(Not run)

Convert an open_clip model to ONNX

Description

Runs dimoose's export_onnx.py (in the package's python folder) to write a model as ONNX image and text encoders that onnxModel() runs without PyTorch. The conversion itself needs PyTorch and open_clip, but only once: the folder it writes can be copied to other computers or uploaded to a Hugging Face repository and used from there.

Usage

exportOnnxModel(
  name = "bioclip",
  pretrained = "openai",
  weights = NULL,
  dir = NULL,
  quantize = c("int8", "fp16"),
  python = NULL,
  run = NULL
)

Arguments

name

The model, as for visionModel(): an open_clip architecture name, "bioclip", "bioclip-2", "hf-hub:<organisation>/<model>" or "local-dir:<folder>". Only models with a vision transformer and open_clip's own CLIP tokenizer can be converted.

pretrained

open_clip's tag for the weights of an architecture.

weights

Local checkpoint file instead of a download.

dir

Folder to write to. By default the model's folder in onnxCache(), where onnxModel() and visionModel(backend = "onnx") find it by the same name.

quantize

Which smaller copies to write as well: "int8", "fp16", both or neither (character(0)).

python

The Python to run the script with; it needs the packages in the package's python/requirements.txt and python/requirements-export.txt. By default, if the uv program is installed, uv provides a Python with those packages (downloading them the first time); otherwise python3.

run

For tests: a function ⁠(command, args)⁠ that runs the script and returns its exit status.

Details

Each file is checked against the PyTorch model as it is written; the script prints the files' sizes and how close their embeddings are.

Value

The folder, invisibly.

Files written

  • dimoose_onnx.json: architecture, image size, patch size, embedding dimensions, preprocessing (resize, interpolation, crop, rescale, mean, std), the file names and sizes of each precision, and the names of the encoders' inputs and outputs.

  • image_encoder.onnx: input pixel_values (⁠n x 3 x S x S⁠, float32); outputs image_embeds (⁠n x d⁠, not normalised) and patch_embeds (⁠n x g*g x d⁠, the projected patch tokens).

  • text_encoder.onnx: input input_ids (⁠n x 77⁠, int64); output text_embeds (⁠n x d⁠, not normalised).

  • tokenizer.json: the tokenizer in the Hugging Face tokenizers format, which transformers.js also reads.

  • with quantize: image_encoder_int8.onnx, text_encoder_int8.onnx (weights as 8-bit integers, about a quarter of the size) and image_encoder_fp16.onnx, text_encoder_fp16.onnx (16-bit floats, half the size, same embeddings).

Examples

## Not run: 
exportOnnxModel("bioclip")
model <- onnxModel("bioclip", precision = "int8")

## End(Not run)

Export an interactive dichotomous key

Description

Writes a dimoose key as a single self-contained HTML page. With JavaScript on, the page steps through the key one couplet at a time, keeps a trail of the choices made (each one clickable to go back to it) and supports the browser's Back button. Without JavaScript, the same page is the full key with every lead hyperlinked to its next couplet or taxon. Each taxon gets a result section listing the characters on its path, and a map of the key shows where the user is (see map).

Usage

exportWizard(
  key = NULL,
  file = NULL,
  displayLinks = TRUE,
  order = "original",
  title = NULL,
  map = TRUE,
  staticPaths = NULL,
  matrix = NULL,
  characters = NULL,
  taxon = "Taxon",
  offline = FALSE,
  photoModel = NULL,
  photoRuntime = NULL
)

Arguments

key

A dimoose object, e.g. from importFishbase(). May be NULL if matrix is given.

file

Path of the HTML file to write. If NULL, the HTML is returned as a string instead.

displayLinks

If TRUE, show the couplet numbers each lead leads to and comes from, as in a printed key. If FALSE, hide them. Leads stay clickable either way.

order

Deprecated and ignored. Couplets are always exported in the key's own order. "parsimony" was never implemented; to ask the questions that are easiest to observe first, build the key with keyFromMatrix() and give the characters a Cost.

title

Page title. Defaults to the key's description.

map

If TRUE, include a map of the key: the whole key drawn as a tree, marking the current couplet, the path taken so far and the branches (and number of taxa) still possible. It sits beside the key on wide screens and behind a Map button on phones.

staticPaths

If TRUE, write each taxon's path through the key into the page, so it shows without JavaScript. If FALSE, the page builds the path in the browser when a result is shown, which keeps large keys (such as phylotreeKey() on the whole tree) much smaller. The default is TRUE for keys with up to 1,000 taxa.

matrix

Optional character matrix: a data frame with one row per taxon, the taxon's name in column taxon and one column per character. Cells are states; several states of one taxon are separated by "/"; NA is unknown or inapplicable. See 'Picking characters in any order'.

characters

Optional data frame describing the characters of matrix: column Character (a column name of matrix) and any of Cost (lower is easier to observe), Season (when it can be observed) and Needs (e.g. "hand lens").

taxon

Name of the column of matrix that holds the taxon names.

offline

If TRUE, also write a web app manifest, a service worker and icons next to file, so that the page can be installed and opened without a connection. Needs file. See 'Offline use'.

photoModel

Optional: a folder written by exportBrowserModel() for this key, or the web address (⁠https://...⁠) of one. Adds "Identify from a photo" to the page. See 'Identifying from a photo'.

photoRuntime

Optional web address of a folder holding the dist files of onnxruntime-web (ort.wasm.min.js, ort-wasm-simd-threaded.mjs and .wasm), to serve them yourself instead of loading them from jsDelivr. Relative addresses are relative to the page.

Details

The key is checked first: leads pointing at missing couplets, terminal leads without a taxon, couplets with fewer than two leads and couplets that cannot be reached from the first one are reported as warnings and marked in the page.

Value

The path to file, invisibly, or the HTML as a character string if file is NULL. With offline = TRUE or a photoModel folder the paths of the other files written are in attribute "files".

Not sure

Every couplet has an "I'm not sure" button (keys u or ⁠?⁠). It follows all leads of the couplet at once. The page then asks the questions of one branch at a time, lists the branches still open (click one to answer it next), and counts and marks on the map the taxa possible in any of them. Each answer narrows its own branch. If the reader stays unsure to the end, the page lists every result that is left. Back undoes one step at a time. The address of the page records the open branches (⁠#open=c-3,c-9⁠), so a reload or a shared link returns to the same place; the trail of choices comes back too when the browser restores the page's history entry.

Picking characters in any order

With matrix, the page gains a second way in: a list of the characters of the matrix, to answer in any order. Picking a state removes the taxa that do not have it. A taxon with several states ("a/b") fits any of them, and a taxon whose state is unknown (NA) fits all. The characters that still separate the remaining taxa are listed first: cheapest first when characters gives a Cost, then those that leave the fewest taxa on average. Taxa of the matrix are matched to the key's taxa by name (ignoring case), so the remaining taxa link to their result sections and show on the map, and the page offers to continue step by step from the couplet they all share. Without JavaScript the matrix is a plain table. The two ways are otherwise independent: answers given step by step do not filter the character list. With key = NULL the page is the character list alone.

Offline use

The page itself needs no connection once it is open, but a browser will not reopen a page it cannot fetch. offline = TRUE writes four files next to file, named after it: a web app manifest (⁠<name>.webmanifest⁠), a service worker (⁠<name>-sw.js⁠) and two icons (⁠<name>-icon-192.png⁠, ⁠<name>-icon-512.png⁠; the package logo if the rsvg package is installed, otherwise a simple drawn icon). The worker stores the page, the manifest, the icons and the figures the page loads from other sites, so that the page can be installed on a phone ("Add to Home Screen") and opened without signal. Upload all five files to the same folder. Limits:

  • Browsers run service workers only on ⁠https://⁠ or ⁠http://localhost⁠. A page opened from disk (⁠file://⁠) works as before but is not installed and stores nothing.

  • The page must be opened once with a connection before it works without.

  • Figures on other sites are fetched when the worker installs and again when first shown; any that fail are skipped. A site that does not allow cross-origin requests gives an opaque copy, which browsers count as several megabytes each against the storage they allow, so a key with many such figures may not store them all. Figures embedded in the key (see imageMeta()) are part of the page and always available. Full-size figures and taxon pages that open in a new tab are not stored.

  • The worker only controls the page it was written for (its scope is the page's own address), so several keys can share a folder. Serve the page under its file name; a server that drops the .html from addresses puts the page outside the worker's scope.

Identifying from a photo

photoModel adds an "Identify from a photo" button. The reader takes or picks a photo; the page loads the model and onnxruntime-web (the program that runs it, version 1.31.1, about 15 MB, from the jsDelivr CDN unless photoRuntime says otherwise), prepares the picture as the model's model.json says, runs the image encoder in the browser and compares the result with the text embeddings of the key's taxa. The photo never leaves the reader's device. The page then marks the likely taxa on the map, lists the five best with their scores, each linked to its result, and offers to narrow the key to them: the leads that reach none of them are greyed and the count of remaining taxa follows, but every lead still works. The key stays the authority; the photo is a hint.

photoModel is a folder written by exportBrowserModel() for the same key, or the web address of one. The model is not put into the page. A folder is copied next to file as ⁠<name>_model/⁠ (so file is needed); upload it with the page. A web address is used as it is; a model on another site must allow cross-origin requests. What to expect:

  • Nothing is downloaded until the reader has chosen a photo, been told the size and agreed. An int8 BioCLIP encoder is 90 MB; with the runtime that is about 105 MB on first use. The page shows the progress.

  • With offline = TRUE the service worker keeps the model and the runtime once they have been fetched, so photos can be identified without a connection afterwards. That is the same 105 MB or so of storage on the reader's device, and the first use still needs a connection.

  • It needs a browser with WebAssembly, and memory: several times the size of the encoder while it loads. If the browser cannot run the model, runs out of memory or cannot fetch it, the page says so and the key works as before. A page opened from disk (⁠file://⁠) cannot load the model.

  • Only the WebAssembly backend is used, on one thread unless the page is served cross-origin isolated. That suits int8 encoders, which WebGPU cannot run.

  • The scores compare the key's taxa with each other and add up to 100%, so a photo of something that is not in the key still gets a confident best match. If model.json has a min_similarity, a best match whose cosine similarity is under it is shown as "Not much like any taxon of this key" without a percentage.

Notes on leads

A Note column in the key's leads, as pruneKey() writes when it collapses a couplet (the character of the couplet that is no longer asked), is shown under its lead as "Also:" and in the path of each result. A pruned_note in the key's meta (the same for a collapsed first couplet) is shown under the title.

Hook for other scripts

The page defines window.dimoose so that another script, for example one that identifies a photo in the browser, can work with the page without knowing how it is built:

  • dimoose.taxa: array of ⁠{ id, name }⁠, one per result.

  • dimoose.highlight(scores, opts): scores is an object mapping taxon names or ids to numbers, or an array of ⁠{ taxon, score }⁠; null clears. Marks those taxa on the map and in every list of candidates and lists the five best above the key, headed by opts.label (default "Suggested"). Scores all within 0 to 1 are shown as percentages. Returns the ids it marked, best first.

  • dimoose.goToTaxon(nameOrId): shows that result; false if unknown.

  • dimoose.narrow(namesOrIds): narrows the key to these taxa (see 'Identifying from a photo'); null undoes it. Returns the ids kept.

  • dimoose.state(): ⁠{ shown, open, remaining, picks, narrowed }⁠, the section shown, the open branches, the ids of the taxa still possible, the states picked in the character list and the ids the key is narrowed to (or null).

  • dimoose.photo (only with photoModel): identify(file), a promise of the ranked taxa, and the pure steps preprocess() and score().

  • dimoose.onChange(fn): calls fn(state) after every step.

See Also

keyFromMatrix() to build a key from a character matrix.

Examples

leads <- data.frame(
  Statement = c("1", "1", "2", "2"), Choice = c("a", "b", "a", "b"),
  Character = c("Wings present", "Wings absent", "Two wings", "Four wings"),
  Next = c("2", "-", "-", "-"), Taxon = c("", "Silverfish", "Fly", "Bee")
)
key <- keyFromLeads(leads, "A tiny key")
html <- exportWizard(key)
substr(html, 1, 15)

# With a character matrix the reader can also pick characters in any order
insects <- data.frame(
  Taxon = c("Silverfish", "Fly", "Bee", "Ant"),
  Wings = c("absent", "present", "present", "present/absent"),
  Sting = c("absent", "absent", "present", NA)
)
characters <- data.frame(Character = c("Wings", "Sting"), Cost = c(1, 3),
                         Needs = c("", "hand lens"))
html <- exportWizard(key, matrix = insects, characters = characters)

# A matrix alone makes a page with only the character list
html <- exportWizard(NULL, matrix = insects, title = "Four insects")

# An installable copy that opens without a connection once it is served
# over https: the page, a manifest, a service worker and two icons
dir <- tempfile("key")
dir.create(dir)
out <- exportWizard(key, file.path(dir, "insects.html"), offline = TRUE)
list.files(dir)
unlink(dir, recursive = TRUE)

Compute a key's feature scores from embeddings

Description

Model-free: give it the embeddings and it applies each feature's rule. centroid_patch_max features need patches; clip_text_pair features need image and textEmb (rows named by prompt). For external features supply the scores yourself.

Usage

featureScores(key, image = NULL, patches = NULL, textEmb = NULL)

Arguments

key

A dimoose key with a features table.

image

⁠objects x d⁠ matrix of image embeddings (unit rows).

patches

⁠objects x patches x d⁠ array of patch embeddings.

textEmb

⁠prompts x d⁠ matrix of text embeddings whose row names are the prompt texts, e.g. from embedTexts().

Value

An ⁠objects x features⁠ matrix with feature ids as column names.

See Also

scoreImages() for images-to-scores in one call.


A table of machine-computable features

Description

Keys built from images (see keyFromTerms() and keyFromClusters()) carry a features table that says how a machine computes the score each lead tests. classify() walks a key over a matrix of such scores.

Usage

featureTable(
  id,
  kind,
  label = id,
  prompt = NA,
  negative_prompt = NA,
  region = NA,
  embedding = NULL,
  exemplars = NULL,
  definition = NA
)

Arguments

id

Feature ids, e.g. "sniglet:dulmzil" or "text:round_headlights".

kind

"centroid_patch_max" (highest cosine similarity between any image patch embedding and embedding), "clip_text_pair" (cosine with prompt minus cosine with negative_prompt) or "external" (computed by the user).

label

Name shown to people; defaults to id.

prompt, negative_prompt

Texts for clip_text_pair features.

region

Crop as "y0,y1,x0,x1" fractions, or NA for the whole image.

embedding

List of numeric vectors (one per feature), or NULL.

exemplars

List of data frames with columns image, patch, score and optionally png (base64) showing what the feature means.

definition

Optional sentence saying what the feature is, shown in the wizard's glossary (see renameSniglets()).

Value

A data frame with the columns above; embedding and exemplars are list columns.

Examples

featureTable("sniglet:x", "centroid_patch_max", embedding = list(c(1, 0, 0)))

Generate a tree from a feature matrix

Description

Fits a classification tree (or tree ensemble) with the taxon column as the response and every other column as a feature.

Usage

generateTree(
  data,
  target_col,
  method = c("rpart", "randomForest", "gbm"),
  params = list()
)

Arguments

data

A data frame containing the feature matrix.

target_col

Name of the target/response column.

method

"rpart", "randomForest" (requires the 'randomForest' package), or "gbm" (requires the 'gbm' package; "multinomial" distribution unless the response is numeric 0/1, which uses "bernoulli").

params

Optional list of arguments passed to the fitting function, overriding the defaults.

Details

The rpart defaults are set for identification data, which usually has one row per taxon: rpart.control(minsplit = 2, minbucket = 1, cp = 0), so the tree keeps splitting until every taxon is separated. (rpart's own defaults, ⁠minsplit = 20, minbucket = 7, cp = 0.01⁠, make no split at all on such a matrix.) With many specimens per taxon pass your own control in params to get a smaller tree, or just the settings to change, e.g. params = list(cp = 0.01) or params = list(maxdepth = 3). keyFromRpart() turns the tree into a key; keyFromMatrix() goes from matrix to key in one step and handles taxa that are variable for a character.

Value

A fitted rpart, randomForest or gbm object. For the last two the call is the call to generateTree().

Examples

features <- data.frame(
  family = factor(c("A", "B", "C")),
  fin = factor(c("x", "y", "y")),
  head = factor(c("p", "p", "q"))
)
fit <- generateTree(features, "family")
fit
keyFromRpart(fit)$leads

Which mutations each haplogroup carries

Description

Follows every haplogroup's path from the root of the tree and records, for each requested site, whether the haplogroup's haplotype differs from the root (RSRS) there. Back-mutations are applied as Haplogrep defines them (they undo the latest change at that site), so a site gained and later reverted on a lineage is not carried below the reversion. Insertions are tracked as their own sites ("573.X", "2491.1"), separate from substitutions at the same position; a deletion counts as carried.

Usage

haplotypeMatrix(
  haplogroups,
  sites = NULL,
  tree = dimoose::phylotree17,
  mutations = dimoose::phylotree17_mutations
)

Arguments

haplogroups

Haplogroup names, one per row of the result; repeats are allowed (e.g. one per sample).

sites

Sites to report: positions as character ("16311") or insertion sites ("573.X"). Defaults to the sites of recurrentPositions().

tree, mutations

The tree and its mutations; default phylotree17 and phylotree17_mutations. tree needs haplogroup, parent and depth; mutations needs haplogroup, notation, position, type, ancestral and derived.

Value

An integer matrix of 0/1, one row per entry of haplogroups (row names are the haplogroups) and one column per site.

See Also

recurrentPositions(), macroHaplogroup()

Examples

haplotypeMatrix(c("H2a2a1", "U5b", "L0a1b"), sites = c("152", "16311", "16278"))

Name what a photo shows, as precisely as the evidence allows

Description

Like suggestSpecies(), compares each photo with the key's own species, but gives one answer per photo: the species when it alone is likely enough, otherwise the genus, family or order whose species together are. A photo of a cottonwood that the model cannot place between the key's two cottonwoods is reported as genus Populus rather than as a coin toss between them.

Usage

identifyPhotos(
  key,
  model,
  paths,
  confidence = 0.8,
  lineage = NULL,
  kingdom = NULL,
  prompt = "a photo of %s."
)

Arguments

key

A dimoose key.

model

A visionModel().

paths

Image files, or an imageSet() (its path column is embedded and its id column names the rows).

confidence

The probability an answer must reach, between 0 and 1.

lineage

Optional data frame with columns Taxon (as in the key), genus, family and order, such as the output of checkNames().

kingdom

Optional kingdom ("Plantae", "Animalia", "Fungi", ...), which helps GBIF tell apart names that are used in more than one kingdom.

prompt

The text each name is put into, with ⁠%s⁠ where the name goes. The default is the one BioCLIP was trained with.

Details

The genus, family and order of each species come from GBIF (see checkNames()), which needs an internet connection, or from a table you give in lineage. Results with no lineage (for example ones named only by a common name) count only as themselves.

Value

A data frame with one row per image: image, name (the scientific name of the answer), rank ("species", "genus", "family" or "order"), probability, confident (whether probability reaches confidence; when nothing does, the answer is the most likely species) and Taxon (the key's result, for species-level answers).

Examples

## Not run: 
model <- visionModel(name = "bioclip-2")
identifyPhotos(montanaKey(), model, "my-tree.jpg", kingdom = "Plantae")

## End(Not run)

Embed images in a key's metadata

Description

Reads image files and returns ⁠image_<id>⁠ rows for a key's meta, which exportWizard() embeds in the page (each image once); leads refer to them by id in their Image column (several separated by ⁠;⁠).

Usage

imageMeta(paths, ids = NULL, maxSize = 1200)

Arguments

paths

Image files.

ids

Names for the images; default the file names without extension.

maxSize

Photos larger than this many pixels on their long side are scaled down to it when the 'magick' package is installed, so keys made from camera photos stay small; without 'magick' they are embedded as they are.

Value

A key/value data frame.


MIME type of an image file from its first bytes

Description

MIME type of an image file from its first bytes

Usage

imageMime(path)

Arguments

path

One file.

Value

"image/png", "image/jpeg", "image/gif", "image/webp" or NA.


A set of labelled images

Description

The starting point for automated key generation: one row per photo, with the class (species, model, ...) the photo shows. A key separates the classes, so there must be at least two.

Usage

imageSet(paths, label = NULL)

Arguments

paths

Image files, or a single folder. A folder is searched (with its subfolders) for image files: png, jpg/jpeg, gif, webp, bmp and tif/tiff.

label

Class of each image:

  • NULL: for a folder whose images are in subfolders, the name of the subfolder each image is in (pics/cardinal/IMG_01.jpg is a cardinal); otherwise the file name without its extension;

  • a character vector as long as paths (for a folder, as long as the files found, which are sorted by path);

  • a function applied to the file names without extensions (e.g. function(x) sub("_.*$", "", x) so that ⁠1967_alt⁠ counts as 1967).

Value

A data frame with id, path and label. id is the file name without its extension; when two files share a name, their folder is put in front (cardinal_IMG_01) so that ids are unique.

Examples

imageSet(c("pics/1967.png", "pics/1967_alt.png"), label = function(x) sub("_.*$", "", x))
imageSet(c("pics/cardinal/1.jpg", "pics/robin/1.jpg"), label = c("cardinal", "robin"))
# the example photos that ship with dimoose: one folder per fruit
pics <- imageSet(system.file("extdata", "pics", package = "dimoose"))
table(pics$label)

Import a dichotomous key from FishBase

Description

Downloads a key from FishBase and converts it into a dimoose object. See https://www.fishbase.se/keys/allkeys.php for the list of keys and their key codes.

Usage

importFishbase(
  keycode,
  fishbaseUrl = "https://www.fishbase.se/",
  separateTerms = TRUE,
  usePhyloService = c("GNA", "GNR", "none", "TNRS"),
  embedImages = FALSE
)

Arguments

keycode

Numeric or character FishBase key code.

fishbaseUrl

Base URL of the FishBase mirror, with a trailing slash.

separateTerms

If TRUE, split each lead's character statement on ";" into one row per term.

usePhyloService

Taxonomic name resolution: "GNA" verifies terminal taxa with the Global Names verifier (taxize::gna_verifier(), requires the 'taxize' package); "none" skips resolution. "GNR" is accepted as an alias for "GNA". "TNRS" is no longer available because taxize::tnrs() is defunct.

embedImages

If TRUE, download the key's morphology images and store them base64-encoded in meta. Image URLs are always stored.

Value

A dimoose object.

See Also

parseFishbase() to build a key from saved HTML pages.

Examples

## Not run: 
sharks <- importFishbase(1)
sharks$taxa

## End(Not run)

Download a balanced sample of iNaturalist photos

Description

Collects field photos to test a key with: for each species, n observations from iNaturalist with an openly licensed photo, the photo files, and a manifest that says where every photo came from, so that a benchmark can be cited and run again.

Usage

inatPhotos(
  taxa,
  n = 20,
  place = NULL,
  qualityGrade = "research",
  licenses = c("cc0", "cc-by", "cc-by-nc"),
  dir,
  fetch = benchInatGet,
  download = benchDownload,
  size = "medium",
  pool = 200,
  strata = 10,
  idBelow = NULL,
  seed = 1,
  perObserver = 1,
  photosPerObservation = 1,
  query = list(),
  pause = 1
)

Arguments

taxa

The species: scientific names, results written "Common name (Genus species)", a dimoose key (its results are used), or iNaturalist taxon ids (a numeric vector, named to give the labels).

n

Observations to sample per species.

place

Optional place to restrict the sample to: an iNaturalist place id, or a name to look up (the best match is used and reported).

qualityGrade

iNaturalist quality grade. "research" means the community agreed on the species.

licenses

Photo licences to accept.

dir

Folder to put the photos and manifest.csv in; one subfolder per species. Keep it outside a package or repository.

fetch

Function ⁠function(url, query)⁠ that performs an API request and returns the parsed JSON as a list. The default calls the iNaturalist API; tests give their own.

download

Function ⁠function(url, destfile)⁠ that saves a photo.

size

Photo size: "small" (240 px), "medium" (500 px), "large" (1024 px) or "original".

pool, strata

Size of the pool the sample is drawn from, and the number of slices of the id range it is collected from.

idBelow

Only observations with an id below this; see above.

seed

Seed for drawing the sample from the pool.

perObserver

Most observations to take from one observer per species.

photosPerObservation

Most photos to take from one observation.

query

Further parameters for the observation search, e.g. list(month = "6,7,8") for summer photos only.

pause

Least number of seconds between requests.

Value

The manifest, a data frame with one row per photo: taxon (the label, as given in taxa), scientific_name, taxon_id, observation_id, photo_id, license, attribution, url, observation_url, observed_on, file (relative to dir), quality_grade, place_id, id_below and retrieved; plus path, the full file name on this computer, which is not written to manifest.csv. benchmarkKey() takes it as its photos.

How the sample is drawn

The sample depends only on the arguments, not on the day it is drawn:

  1. idBelow fixes the snapshot: only observations with a smaller id are considered. When it is NULL the newest observation id at the time of the call is used and recorded in the manifest (id_below); give that value to draw the same sample again later.

  2. For each species a pool of up to pool observations is collected from strata equal slices of the id range (the newest observations below each cut), so that the pool spans the years rather than being last month's photos. Ids follow upload dates, so each slice is a short stretch of time and the same stretches are used for every species; raise strata (one request each) for an evener spread over the seasons.

  3. From the pool, sorted by id, n observations are drawn with seed, taking at most perObserver from one observer, so that one person's many photos of the same tree cannot dominate.

Observations that were deleted or lost their research grade since are simply no longer returned, so an old sample can shrink; the photo files already in dir are kept and not downloaded again. A photo that cannot be downloaded, or that turns out not to be an image, is left out of the manifest with a warning; it is not replaced by another.

Which taxon a name means

A name is looked up in iNaturalist's taxon search and must match a taxon's name exactly; a looser match (the search also finds "Oenantheae" and "Oenanthe oenanthe" for "Oenanthe") is never used. A name iNaturalist lists as a synonym resolves to the taxon it now belongs to, with a message. The same name can belong to two taxa (the genus Oenanthe is a bird and a plant): then the one whose rank fits the name best and that has the most observations is used, with a warning that lists the candidates and their ids. Give the taxon id instead of the name to choose yourself. The manifest records the id that was used.

API etiquette

Requests are spaced at least pause seconds apart (iNaturalist asks for about one request a second) and identify themselves with a generic User-Agent. Only photos whose own licence is in licenses are taken; the manifest keeps each photo's licence and attribution, which the CC BY licences require you to show wherever the photos are.

See Also

benchmarkKey()

Examples

## Not run: 
photos <- inatPhotos(montanaKey(), n = 20, dir = file.path(tempdir(), "inat"))
table(photos$taxon)

## End(Not run)

Invent pronounceable names

Description

Random words made of consonant onsets, vowels and codas. They name sniglets: visual features that have no English name (as fordera does).

Usage

inventNames(n, seed = 1)

Arguments

n

Number of names.

seed

Random seed, so the same call gives the same names. The global random number generator state is left unchanged.

Value

A character vector of n distinct lower-case words.

Examples

inventNames(5)

Draw a key as a box diagram

Description

Draws a dimoose key top-down as an SVG picture, the way keys are often drawn on paper: each couplet is a blue box holding its question, its leads are blue boxes below it, and every lead either continues to the next couplet or ends in a green box naming the result. A result written as "Common name (Genus species)" is drawn on two lines with the scientific name in italics.

Usage

keyDiagram(key, file = NULL, images = TRUE, title = NULL)

Arguments

key

A dimoose key.

file

Optional file to write: .svg, or .png and .pdf when the 'rsvg' package is installed.

images

If TRUE, show a thumbnail in a result's box when the terminal lead names an image (Image column) that is embedded in the key's meta (see imageMeta()).

title

Heading drawn above the tree; by default the key's desc. Use NA for none.

Details

Thumbnails are shrunk to 112 pixels when the 'magick' package is installed, which keeps diagrams of keys with large photos small. When 'rgnparser' and its 'gnparser' program are installed, only text that parses as a scientific name is set in italics.

A couplet reached from more than one lead is drawn once, under the first of them; the other leads get a dashed line to it.

Value

The SVG as a single string, invisibly when file is given.

See Also

exportWizard() for an interactive version of the key.

Examples

keyDiagram(arachnidaKey(), tempfile(fileext = ".svg"))

Build a key from a multiple sequence alignment

Description

One command from aligned sequences to a dichotomous key. A phylogeny program builds a tree from the alignment, the tree is rooted and made strictly bifurcating, and every fork becomes a couplet whose two leads are the alignment sites that tell its two clades apart ("152C 16311T" against "152T 16311C"). Each sequence is a result.

Usage

keyFromAlignment(
  alignment,
  tool = c("nj", "bionj", "upgma", "parsimony", "ml", "fasttree", "iqtree"),
  tree = NULL,
  outgroup = NULL,
  reference = NULL,
  type = c("auto", "DNA", "AA"),
  model = NULL,
  maxSites = 3,
  exe = NULL,
  args = NULL,
  desc = "Key generated from a sequence alignment",
  meta = NULL
)

Arguments

alignment

The alignment: a FASTA file, an ape DNAbin/AAbin object (e.g. from ape::read.dna() for Clustal or PHYLIP files), a phangorn phyDat, a character matrix with one named row per sequence and one column per site, or a named character vector of equal-length strings.

tool

The program that builds the tree:

  • "nj", "bionj": neighbour joining in ape;

  • "upgma", "parsimony", "ml": UPGMA, the parsimony ratchet and maximum likelihood in phangorn;

  • "fasttree", "iqtree": the FastTree or IQ-TREE command-line program, which must be installed.

tree

A tree you built yourself, instead of tool: an ape phylo object, a Newick file or a Newick string, with the sequence names as tip labels. It may cover only some of the sequences, and may have polytomies. This is the way to use any other phylogeny program.

outgroup

Name(s) of the sequence(s) to root on. Without it, a tree that is not already rooted is rooted at its midpoint. Sequences can be named in full or by the first word of their FASTA header.

reference

Name of a sequence in the alignment to number sites by (its ungapped positions; columns where it has a gap are numbered "315.1", "315.2"). By default sites are alignment columns.

type

"DNA", "AA", or "auto" to guess from the sequences.

model

Substitution model, passed to the tool: the model of ape::dist.dna() for the distance methods (default "K80"; "JTT" through phangorn::dist.ml() for proteins), the model of phangorn::pml_bb() for "ml" (default "GTR+G(4)", or "LG+G(4)" for proteins), or IQ-TREE's -m (default: IQ-TREE chooses).

maxSites

Most sites listed in a lead.

exe

Path to the FastTree or IQ-TREE program, if it is not on the PATH.

args

Extra command-line arguments for FastTree or IQ-TREE.

desc, meta

Title and extra metadata of the key.

Details

Sites are chosen per fork: a site is diagnostic when the states seen on one side are never seen on the other. Up to maxSites diagnostic sites are listed, those with the least missing data first. If a fork has no diagnostic site, the best-separating one is used and marked (most). If no site separates the two sides, the sequences below it are split by their sites alone, whatever the tree says. Sequences that no site tells apart share a result ("A / B"), as do identical sequences. Gaps and ambiguity codes are treated as missing data.

The first site of each couplet is also a machine-readable test, so classify() can place sequences scored with alignmentScores().

Value

A dimoose key. Its meta records the tool, the rooting, the tree in Newick format (newick, names quoted where Newick needs it), how many couplets lack a diagnostic site (weak_couplets) and how many do not follow the tree (regrouped_couplets).

See Also

alignmentScores(), phylotreeKey(), toNewick()

Examples

aln <- c(human = "ACGTACGTACGA", chimp = "ACGTACGTATGA", mouse = "ACCTACATGCGA",
         rat   = "ACCTACATGCGT", fish  = "TCCTTCGTACCA")
if (requireNamespace("ape", quietly = TRUE)) {
  key <- keyFromAlignment(aln, outgroup = "fish")
  key$leads[, c("Statement", "Character", "Next", "Taxon")]
  classify(key, alignmentScores(key, aln))
}

# the example files that ship with dimoose
fasta <- system.file("extdata", "aligned.fasta", package = "dimoose")
mtdna <- system.file("extdata", "mtdna.fasta", package = "dimoose")
if (requireNamespace("ape", quietly = TRUE)) {
  keyFromAlignment(fasta)
  keyFromAlignment(fasta, tree = system.file("extdata", "aligned.nwk", package = "dimoose"))
  keyFromAlignment(mtdna, reference = "H2a2a1", outgroup = "L0a1")
}
## Not run: 
keyFromAlignment(mtdna, tool = "iqtree", outgroup = "L0a1")   # needs IQ-TREE installed

## End(Not run)

Build a key by clustering classes and describing each split

Description

Classes are clustered on the mean embedding of their images (Ward's method), giving a binary tree. Each split is then described by the vocabulary pair that best separates its two sides: the pair maximising |(L.yes - L.no) - (R.yes - R.no)| over the side means, not used above that split. The "Has feature" lead goes to the side the feature scores higher on. Machine answers use the same two prompts.

Usage

keyFromClusters(
  image,
  images,
  vocabulary,
  textEmb,
  template = "{x}",
  order = NULL,
  maxPerSide = 10,
  calibrate = c("none", "midpoint"),
  desc = "Key generated from image clusters",
  meta = NULL
)

Arguments

image

⁠images x d⁠ matrix of unit-length image embeddings.

images

imageSet() data frame, one row per row of image.

vocabulary

As in vocabularyPrompts().

textEmb

Text embeddings of vocabularyPrompts(vocabulary, template), rows named by prompt (as embedTexts() returns them).

template

Must match the one used to make textEmb.

order

Optional function of a character vector of labels returning a number; at each split the child with the smaller value comes first.

maxPerSide

Most images per side used for the side means.

calibrate

"none" (threshold 0) or "midpoint" (halfway between the two sides' mean scores).

desc, meta

Title and metadata.

Value

A dimoose key with clip_text_pair features.


Build a dimoose key from a table of leads

Description

The common constructor behind sharkKey(), arachnidaKey(), vibrioKey() and phylotreeKey(): takes one row per lead and returns a dimoose object.

Usage

keyFromLeads(leads, desc = NA, meta = NULL, taxa = NULL, features = NULL)

Arguments

leads

Data frame with columns Statement (couplet), Choice, Character (the lead's text), Next (next couplet, or "-" if the lead ends in a result) and Taxon (the result, for terminal leads). Optional columns Label, Image, ImageLink and TaxonUrl are used by exportWizard(); any others are kept.

desc

Title or description of the key.

meta

Key/value data frame of metadata (e.g. citation).

taxa

Data frame of results with submitted_name and matched_name; by default every taxon matches itself.

features

Optional featureTable() for keys a machine can follow (the leads then also carry Feature, Test and Threshold).

Value

A dimoose object.

Examples

leads <- data.frame(
  Statement = c("1", "1", "2", "2"), Choice = c("a", "b", "a", "b"),
  Character = c("Wings present", "Wings absent", "Two wings", "Four wings"),
  Next = c("2", "-", "-", "-"), Taxon = c("", "Apterygota", "Diptera", "Other")
)
k <- keyFromLeads(leads, "A tiny key")
k
k$toNewick()

Build a dimoose key from a character matrix

Description

Generates a dichotomous key from a taxon-by-character matrix, the form most identification data is kept in (DELTA, Lucid, SDD, a spreadsheet of test results). Unlike a classification tree, the key is built the way a person writes one: a taxon that is variable for a character is keyed out down both leads of every couplet that uses the character.

Usage

keyFromMatrix(
  matrix,
  taxon = "Taxon",
  characters = NULL,
  season = NULL,
  variable = c("both", "missing"),
  desc = "Key generated from a character matrix",
  meta = NULL,
  maxCouplets = 10000
)

Arguments

matrix

A character matrix: a data frame with one row per taxon (see 'The matrix'), e.g. the matrix of readDELTA() or readSDD().

taxon

Name of the column holding the taxon names.

characters

Optional data frame of character metadata with a column Character (matching column names of matrix) and any of Cost (positive number, lower means easier to observe; missing or blank means 1), Season (seasons or conditions in which the character can be observed, separated by ⁠,⁠, ⁠;⁠ or /; blank, "all", "all year" or "always" means always), Needs (free text such as "hand lens", copied to the leads), Type (see 'Numbers and ranges') and InapplicableIf (see 'How the key is built'). The characters table of readDELTA(), readSDD(), readLucid() or readXper() can be passed as it is. Characters not listed cost 1 and can always be observed; rows for characters that are not in matrix are ignored (with a warning if no row matches). If a character has several rows the first is used, with a warning.

season

Optional character vector. Only characters whose Season is blank or includes one of these (ignoring case) are used, e.g. season = "winter" for a bark and twig key.

variable

"both" (default) sends a variable taxon down both leads; "missing" treats variable cells as unknown and fits a classification tree.

desc

Title of the key.

meta

Key/value metadata data frame.

maxCouplets

For variable = "both": give up with an error once the key has more couplets than this (see 'Size of the key').

Value

A dimoose object. With variable = "both" its leads also carry Question (the character's name), MatrixCharacter and MatrixStates (the character and the /-separated states the lead accepts) and, if characters has it, Needs.

The matrix

A plain data frame (or a character matrix with column names) with one row per taxon: a taxon column and one column per character. Cells are state strings (anything else is converted with as.character()). A taxon that is variable (polymorphic) for a character lists its states separated by /, e.g. "+/-" or "red/white". NA or an empty cell means unknown or inapplicable. If a taxon has several rows, they are merged and the taxon takes every state seen in any of them. Characters with fewer than two states among the taxa are dropped. Every column other than taxon is taken as a character, so leave out columns of notes.

Numbers and ranges

A key can only ask about states a specimen either shows or does not, so a measurement is used through the ranges that do not overlap. In a column whose cells are all numbers or ranges, as the readers readDELTA(), readSDD(), readLucid() and readXper() write them ("3", "4-6", "(1-)2-8(-12)", taken from the lowest to the highest number in the cell), ranges that overlap or touch are merged into one state named by its extremes: "3-5", "5-7" and "12-20" give the two states "3-7" and "12-20". A character whose ranges all overlap has one state and is dropped. A message says what was merged or dropped. The cells may end in a unit if it is the same in every cell of the column ("2-8 mm", "5-12 mm"). Negative numbers are not understood (a - is read as "to"). To key a taxon out on part of an overlapping range, cut the measurement into classes yourself and enter the classes as states ("3/5" and "5/7" leaflets in the example below).

If characters has a column Type, as the tables of those readers do, characters of type "text" are not used, the rule above applies to "integer" and "real" characters only, and "categorical" and "ordered" characters are always taken as states.

How the key is built (variable = "both")

Every couplet tests one state of one character against the rest: the first lead reads "Character: state", the second "Character: other or another", naming the other states the taxa still in play may show. A character with three or more states is therefore keyed out over several couplets, and every couplet has exactly two leads.

A taxon follows the first lead if it can show the state and the second if it can show any other state not already ruled out, so a variable taxon follows both. So does a taxon whose state is unknown (NA), since it cannot be ruled out on either side. The guarantee is: whichever of its states a specimen shows for each character, following the key ends at a result that names its taxon.

At each step the couplet is chosen like this. For every state of every character, count the taxa that follow the first lead only (a) and the second lead only (b). The couplet separates a * b pairs of taxa; taxa that follow both leads separate nothing. The split with the largest a * b / Cost is taken; ties go to the character that comes first in the matrix, then to the state that appears first in its column. With no Cost, every character costs 1. A character with Cost 2 is thus chosen over one with Cost 1 only if it separates more than twice as many pairs.

If no split separates a pair, the taxa that are left overlap: one shows "+" and another "+/-", or one has 3 to 5 leaflets and another 5 to 7. A couplet can still rule taxa out on one side ("-" can only be the second taxon), so the split with the largest (a + b) / Cost is taken, provided some taxon is known to show the state and some taxon is known to show another. When no such split is left either, the remaining taxa share a result, written "A / B": with the states observed they cannot be told apart.

A group of taxa that is reached by several routes with the same states still open is keyed out once, and couplets that come out the same lead for lead are merged, so a couplet may be the target of more than one lead, and a variable taxon appears at more than one result.

If characters has a column InapplicableIf ("Other character=state" or "Other character=state/state", several rules separated by ⁠;⁠, as written by readSDD() and readDELTA()), a character is not asked about while a taxon still in play may show one of those states of the other character: the key asks whether there is an aerial stem before it asks whether the stem is hairy. The rule is dropped where it would leave no couplet at all.

Size of the key

Because variable and unknown taxa follow both leads, the key grows with the number of such cells. Random two-state matrices of 200 taxa and 60 characters gave 199 couplets with no unknown cells, about 550 with 30% unknown and about 3400 with 50%. A matrix in which most characters are known for only a few taxa can need a number of couplets that doubles with every character; maxCouplets stops such a key with an error rather than running out of memory, and a key with more than ten couplets per taxon comes with a warning. Leave out the characters that are unknown for most taxa, or use variable = "missing".

variable = "missing"

Variable cells are treated as unknown and the key is the classification tree fitted by rpart::rpart() (through keyFromRpart()), with Cost passed as rpart's cost. Each taxon then appears at one result, but rpart places a taxon it has no state for with surrogate splits, so a specimen showing the taxon's other state is keyed out wrongly. A taxon rpart cannot place at a split at all is listed at every result below that split (a message says how many such taxa there are). The key is smaller; use it when that matters more than the guarantee above. A lead may name several states ("Colour: red,white"). InapplicableIf is not used.

See Also

vibrioKey(), vibrioMatrix(), keyFromRpart(), generateTree()

Examples

# Some of Montana's broadleaf trees (see montanaKey() for the key written
# by hand). Box elder has 3 to 5 leaflets and green ash 5 to 7, so a leaf
# with 5 could be either.
trees <- data.frame(
  Taxon = c("Box elder", "Green ash", "Rocky Mountain maple", "Bur oak",
            "Quaking aspen", "Paper birch", "Water birch"),
  `Buds and leaves` = c("opposite", "opposite", "opposite", "alternate",
                        "alternate", "alternate", "alternate"),
  `Leaf type` = c("compound", "compound", "simple", "simple", "simple",
                  "simple", "simple"),
  Leaflets = c("3/5", "5/7", NA, NA, NA, NA, NA),
  `Leaf lobes` = c(NA, NA, "lobed", "lobed", "not lobed", "not lobed",
                   "not lobed"),
  Bark = c(NA, NA, NA, NA, "smooth and pale", "white and peeling",
           "gray to brown"),
  check.names = FALSE
)
k <- keyFromMatrix(trees, desc = "Some Montana trees")
k$leads[c("Statement", "Choice", "Character", "Next", "Taxon")]

# Leaves are easy to see but only there in summer; bud arrangement needs a
# closer look but works all year
characters <- data.frame(
  Character = c("Buds and leaves", "Leaf type", "Leaflets", "Leaf lobes", "Bark"),
  Cost = c(2, 1, 1, 1, 1),
  Season = c("", "summer", "summer", "summer", ""),
  Needs = c("a twig in the hand", "", "", "", "")
)
keyFromMatrix(trees, characters = characters)$leads$Character

# A winter key: what bark and buds cannot separate shares a result
winter <- keyFromMatrix(trees, characters = characters, season = "winter")
winter$leads[c("Character", "Taxon")]

# A matrix read from a DELTA data set, with its character table. Body
# length is used because the three ranges do not overlap; the comments are
# text and are left out.
beetles <- readDELTA(system.file("extdata", "tdwg", "beetles-delta", package = "dimoose"))
keyFromMatrix(beetles$matrix, characters = beetles$characters)$leads[c("Character", "Taxon")]

Build a dimoose key from a classification tree

Description

Converts an rpart::rpart() classification tree into a dichotomous key: every split becomes a couplet with two leads, and every leaf a result. Taxa that the tree cannot separate share a leaf and are listed together ("A / B"). A taxon rpart could not send down either side of a split, because the split's variable is missing for it, is listed at every result below that split.

Usage

keyFromRpart(
  fit,
  desc = "Key generated from a classification tree",
  meta = NULL
)

Arguments

fit

An rpart object with method = "class", e.g. from generateTree().

desc

Title of the key.

meta

Key/value metadata data frame.

Value

A dimoose object.

See Also

keyFromMatrix(), which builds a key straight from a character matrix and handles variable taxa; vibrioKey()

Examples

d <- data.frame(
  taxon = factor(c("A", "B", "C")),
  wings = factor(c("+", "-", "-")),
  spots = factor(c("-", "+", "-"))
)
fit <- generateTree(d, "taxon")
keyFromRpart(fit, "Three taxa")$leads

Build a key from sniglet scores

Description

Each step asks whether the image has one sniglet. method = "balanced" is fordera's algorithm: a sniglet is present when its score is above the quantile of that sniglet's scores over all images; a label has a sniglet if any of its images does; each split uses the unused sniglet that divides the remaining labels most evenly (ties to the earlier one). method = "rpart" suits several images per label. It first gives each sniglet one threshold, the cut of its scores that best separates the labels, and then grows a classification tree on has/lacks, so a sniglet means the same thing wherever the key asks about it.

Usage

keyFromSniglets(
  scores,
  images,
  sniglets,
  method = c("balanced", "rpart"),
  quantile = 0.6,
  desc = "Key generated from sniglets",
  meta = NULL
)

keyFromTerms(scores, images, terms, ...)

Arguments

scores

⁠images x sniglets⁠ matrix from snigletScores().

images

imageSet() data frame with one row per row of scores.

sniglets

The discoverSniglets() object the scores were computed with. Names given with renameSniglets() are used in the key.

method

"balanced" or "rpart".

quantile

Presence threshold quantile for "balanced".

desc, meta

Title and metadata of the key.

terms, ...

keyFromTerms() is the earlier name of keyFromSniglets(); its third argument was called terms.

Value

A dimoose key whose leads carry Feature, Test, Threshold and Question, and whose features table holds the centroids. Feature ids are "sniglet:<coined word>" and never change; label is the name shown, which renameSniglets() changes.

Both leads describe something

A lead that only says a specimen lacks something gives the reader nothing to look for. Where it can, the "Lacks" lead therefore also names what its images have instead: another sniglet that at least 90% of them have and at most 10% of the other side do ("Lacks gaithiark; has blienvaurk"). The machine still tests only the first sniglet. Every lead also records example regions from the images that take it (Examples), which patchExemplars() turns into crops: for a "Lacks" lead with no such second sniglet, they are the regions that come closest to the missing one.


Build a key from a spreadsheet

Description

Reads a key written in a spreadsheet and saved as CSV (in Excel or Google Sheets: File > Save as / Download > CSV). Each row is one lead:

Usage

keyFromSpreadsheet(
  file,
  photos = NULL,
  desc = NA,
  meta = NULL,
  checkNames = FALSE
)

Arguments

file

Path of the CSV file (UTF-8).

photos

Folder holding the photos named in the Photo column; by default the folder the CSV file is in.

desc

Title of the key.

meta

Optional key/value metadata data frame (e.g. a citation row); the photos are added to it.

checkNames

If TRUE, check the species names against GBIF with checkNames() and warn, by spreadsheet row, about names that are misspelled, outdated synonyms or unknown. Needs an internet connection.

Details

  • Couplet: the couplet's number.

  • Description: what the lead says ("Leaves compound.").

  • ⁠Goes to⁠: the couplet this lead leads to, or blank when the lead ends in a species.

  • Species: the result, for leads that end the key. Write it as "Common name (Genus species)" to have keyDiagram() set the scientific name in italics.

  • Choice (optional): the lead's letter; by default the leads of each couplet are lettered a, b, c in the order they appear (after z come aa, ab and so on).

  • Question (optional): what the couplet asks ("Leaf type?"), shown by keyDiagram() and exportWizard().

  • Photo (optional): file names of photos to show with the lead, separated by commas or semicolons; the extension may be left out. A web address (⁠https://...⁠) is kept as a link to the picture instead.

  • Note and Label (optional): a remark on the lead, as pruneKey() writes, and a label to show for the lead in place of couplet and letter. Both are kept in the key's leads as they are.

Column names are matched loosely (case, spaces and punctuation do not matter) and dimoose's own names work too: Statement, Character, Next, Taxon and Image. Other columns are ignored and blank rows are skipped. The photos are embedded in the key, so the wizard and the diagram work offline.

The file may be as Excel saves it: with or without a byte-order mark, in UTF-8 or Windows-1252, and with semicolons between the cells, which is what Excel writes where the comma is the decimal mark.

The Montana key that comes with dimoose is a ready-made example to copy: system.file("extdata", "montana", "montana-key.csv", package = "dimoose"), with its photos in the same folder.

Value

A dimoose key.

See Also

keyToSpreadsheet() to write a key as such a spreadsheet, keyFromLeads() to build a key from a data frame, keyDiagram() and exportWizard() to show it.

Examples

csv <- system.file("extdata", "montana", "montana-key.csv", package = "dimoose")
k <- keyFromSpreadsheet(csv, desc = "Native broadleaf trees of Montana")
k
keyDiagram(k, tempfile(fileext = ".svg"))

Read a key written as plain text

Description

Most identification keys exist only as printed text: floras, monographs, scans in the Biodiversity Heritage Library. keyFromText() reads such a key, one lead at a time (couplet, lead, text and destination), builds a dimoose key with keyFromLeads() and checks it with ⁠$validate()⁠. parseTextKey() is the lower-level half that returns the lead table and the list of problems without building the key.

Usage

keyFromText(text, desc = NA, meta = NULL, llm = NULL, strict = FALSE, ...)

Arguments

text

The key: a file path, one string holding the whole key, or a character vector of lines. Give the key alone, without the descriptions before and after it.

desc

Title of the key. By default the first line above the first lead is used, if there is one.

meta

Key/value data frame of metadata (e.g. citation), passed to keyFromLeads().

llm

Optional function for a first pass over messy text; see the section below.

strict

If TRUE, stop when anything had to be repaired or any problem was found, instead of warning.

...

Further arguments for parseTextKey() (layout, skip).

Details

Layouts that are recognised:

  • Bracketed keys, where the leads of a couplet stand together: ⁠1a.⁠/⁠1b.⁠, 1./⁠1'.⁠ (primes), 1./1., 1/- (a dash for the second lead), 1(a)/1(b), (1)/(1), Roman numerals II., A./AA. or a./aa. lettering, and labels with a back-reference such as ⁠3(1).⁠ (couplet 3, reached from couplet 1). The second lead may also be marked +, or carry no label at all: a new paragraph, as in many older entomological keys, or a line of its own with dot leaders under a complete lead.

  • Indented (yoked) keys, where the second lead of a couplet comes much later and a lead without a destination leads to the couplet printed under it.

  • Indented keys without any couplet numbers, as in most older floras: the leads of a couplet share an indent, and a lead ends in a name, often with its number in the flora (⁠... long-acuminate. 1. S. nigra.⁠). The couplets are made from the indentation and numbered in order. Plain OCR text usually loses the indentation. The couplets are then worked out from the order of the leads, which works when every couplet has exactly two leads: a lead without a name opens the next couplet, a lead with a name is followed by the second lead of the nearest couplet still lacking one. This is reported as a repair; if the leads do not pair off, the key is refused with the line where the pairing breaks. One key at a time: a second heading in the text stops the reading there.

  • Paired keys, where every lead has its own number and names its alternative in parentheses (⁠1 (8) ...⁠ and further down ⁠8 (1) ...⁠). The two are put together as one couplet, named after the lower number.

  • Couplets with three or more leads, and couplets reached from several leads.

The destination is whatever follows dot leaders, a tab, a run of three or more spaces, a colon before a number (lobed: 5) or a spaced dash (⁠a large tree -- Quercus alba⁠): a couplet number (also ⁠go to 5⁠, ⁠see 5⁠, (5), ⁠> 5⁠) or a taxon. A taxon may carry an authority, a common name (in parentheses, or in front of the scientific name) and a page number (⁠Salix, p. 212⁠, ⁠HEMIPTERA (Page 76)⁠, ⁠Culicidae, 96⁠); the genus may be abbreviated (⁠S. alba L.⁠). These go to the Authority, CommonName and Page columns of the lead table. A number in parentheses after a name (⁠Pinus. (7).⁠) is taken for its page or list number, not for a couplet. Lead text may wrap over several lines, with the destination on the last one; words hyphenated at a line end are joined, and blank lines, page numbers, running heads and figure captions in between are skipped, as are footnotes. Text in two columns side by side (as pdftotext -layout gives it) is read column by column; this is reported as a repair. Markdown emphasis and tables are read as plain text.

Scanned text is repaired where that is safe: l, I or | read as 1 and O as 0 in couplet numbers, labels that lost their full stop, labels garbled beyond reading where only one couplet can be meant (V0. between 9 and 11), and destinations that lost their dot leaders (⁠... each year. 11.⁠, ⁠... leaves flat. Abies.⁠). Text with two or more spaces between all words, as many scans have, is recognised, and only clearly wider gaps count as gaps there. Every repair is reported, and with strict = TRUE the function stops instead of repairing.

Nothing is dropped silently. A lead without a destination, a couplet with one lead, a destination couplet that does not exist, a destination that is probably a broken number (S for 5), a jump in the couplet numbers, a label that stands alone on its line, text that looks like a lead without a label and similar faults are reported with the line they are on. keyFromText() gives one warning that lists them and attaches the full table to the key as attr(key, "problems"). Text that was passed over (running heads, descriptions between leads) is listed there as notes.

What a scan has lost cannot be put back: a couplet number misread as another number that also exists in the key gives a wrong link without a warning, and leads whose labels the scan put in a column of their own are reported as lost, not recovered. Check a key read from a scan against the page. Not handled: keys without numbers that are not indented and have couplets of three or more leads, keys that mix letters and unnumbered leads (Gray's Manual), keys divided by section headings that stand for leads (A., *, ⁠**⁠ in Britton & Brown), and text in more than two columns.

Value

A dimoose key. attr(key, "problems") is a data frame of problems, repairs and notes (see parseTextKey()).

A first pass by a language model

For badly scanned keys, llm can be a function that takes the whole text as one string and returns either cleaned-up key text (one lead per line, in any layout above), a data frame with columns couplet, lead, text and destination, or the same as JSON (an array of leads; needs the jsonlite package). Markdown code fences around the answer are taken off. You write that function; dimoose calls no model or API itself. What comes back is not trusted: it goes through the same checks, and each lead is compared with the source, so a lead whose words or taxon do not occur in the source, a destination that is not on the lines of its lead in the source (two names swapped, a lead sent to another couplet), or a lead the model left out, is reported with the line of the source it belongs to (a lead that is not in the source at all has no such line; the message then names the line or row of the model's answer). The title is taken from the source, not from the answer. An answer that cannot be read (cut-off JSON, prose without leads, an empty table) is an error; nothing is taken from it.

See Also

parseTextKey(), keyFromLeads()

Examples

# A bracketed key with dot leaders, wrapped leads and authorities
file <- system.file("extdata", "textkeys", "trees-bracketed.txt", package = "dimoose")
cat(readLines(file, n = 8), sep = "\n")
trees <- keyFromText(file)
trees
trees$leads[, c("Statement", "Choice", "Next", "Taxon", "CommonName")]

# An indented key, given as a string
conifers <- keyFromText(system.file("extdata", "textkeys", "conifers-indented.txt",
  package = "dimoose"))
conifers$toNewick()

# Problems are reported with their line and attached to the key
broken <- "1a. Wings present .... 2
1b. Wings absent .... Apterygota
2a. Two wings .... Diptera
2b. Four wings"
k <- suppressWarnings(keyFromText(broken, "Insects"))
attr(k, "problems")[, c("Line", "Lead", "Kind", "Message")]

## Not run: 
# A first pass by a language model. Any function(text) will do; this one
# calls a chat API and asks for one lead per line.
askModel <- function(text) {
  prompt <- paste(
    "This is a scanned identification key. Rewrite it with one lead per",
    "line as '<couplet><letter>. <text> .... <destination>'. Do not add,",
    "drop or reword leads.\n\n", text
  )
  response <- httr::POST(
    "https://api.example.com/v1/messages",
    httr::add_headers(Authorization = paste("Bearer", Sys.getenv("MODEL_API_KEY"))),
    body = list(model = "some-model", messages = list(list(role = "user", content = prompt))),
    encode = "json"
  )
  httr::content(response)$content[[1]]$text
}
key <- keyFromText("scanned-key.txt", llm = askModel)
attr(key, "problems")

## End(Not run)

How quickly a key gets to a result

Description

Counts the steps (couplets answered) it takes to reach each result of a key and compares the average with the fewest steps any key could need.

Usage

keyQuality(key, weights = NULL)

Arguments

key

A dimoose key.

weights

How common each taxon is: a numeric vector whose names are the key's results as written in its Taxon column, such as the output of occurrenceWeights(). They need not sum to 1, and names that are not in the key are ignored. Taxa that are missing from it or NA count as 0, with a warning. A vector without names is an error, as is a name that is there twice. NULL (the default) treats all taxa as equally common.

Details

A key is a code: each result is spelled by the leads chosen on the way to it. bound is the expected number of steps of the best key there could be for the same taxa and the same weights, among keys whose couplets have no more leads than the largest couplet of this key (maxLeads). It is found by Huffman's construction, which joins the rarest taxa first, so it is a number of steps that a key can really reach: 3 for eight equally common taxa and couplets of two leads, about 3.55 for eleven. No key can do better, however its characters are chosen. efficiency is bound / expectedSteps: 1 for a key that could not be shorter, and lower the more steps the key takes than it would have to. Real keys rarely reach 1, because the characters that would split the taxa evenly may not exist.

When some taxa are met more often than others (weights), a good key asks few questions before reaching the common ones, and the bound takes the weights into account in the same way.

In a key that has a few large couplets among many small ones (a PhyloTree key has couplets of 2 and of 87 leads), the best key is allowed the largest couplet everywhere, so the bound is far below what the key does and efficiency is low. Three more numbers describe such keys without that comparison. entropy is the Shannon entropy of the weights in bits (log2 of the number of taxa when they are equally common): the information a key has to gather. A couplet with k leads can give up to log2(k) bits, and bitsPerStep is the average of that over the steps this key takes (1 for a key of two-lead couplets). capacityUsed is entropy / (bitsPerStep * expectedSteps): the share of the information its couplets could give that the key needs. It is 1 when every couplet splits what reaches it evenly, whatever its size, and it is not a comparison with a shorter key.

Two cases need a rule:

  • A taxon that keys out at several terminal leads counts once: its steps are the mean over its leads, as if its specimens were split evenly among them, and its weight is not multiplied by the number of leads.

  • A couplet reached from several leads (a reticulate key) is counted by its shortest route from the first couplet, so steps is the fewest steps in which a result can be reached. depth is the longest route.

Leads in couplets that cannot be reached from the first couplet, and terminal leads that name no taxon, are left out of the averages.

Value

A list with

  • taxa: a data frame with one row per taxon: Taxon, leads (how many terminal leads end in it), steps (mean over those leads), minSteps, maxSteps and weight (its share of the weights, summing to 1);

  • leads: a data frame with one row per terminal lead: Lead (its label, e.g. "7b"), Statement, Choice, Taxon and steps;

  • meanLeadSteps: mean steps over terminal leads;

  • meanTaxonSteps: mean steps over taxa;

  • expectedSteps: mean steps over taxa weighted by weight (the same as meanTaxonSteps without weights);

  • bound: the expected steps of the best possible key to these taxa with these weights and couplets of at most maxLeads leads;

  • efficiency: bound / expectedSteps, above 0 and at most 1 (NA for a key to a single taxon, which needs no steps);

  • entropy: the Shannon entropy of weight, in bits;

  • bitsPerStep: the most a step of this key can tell, on average (1 when every couplet has two leads);

  • capacityUsed: entropy / (bitsPerStep * expectedSteps), at most 1;

  • maxLeads: the number of leads in the key's largest couplet;

  • depth: the longest route to a result, in steps.

See Also

summary() on a key, which reports the aggregates; occurrenceWeights() for weights from GBIF.

Examples

q <- keyQuality(arachnidaKey())
q$taxa
q$expectedSteps
q$bound
q$efficiency

# spiders and mites are met far more often than the other orders
common <- c(Araneae = 50, Acari = 40, Opiliones = 5, Scorpiones = 3)
suppressWarnings(keyQuality(arachnidaKey(), common))$expectedSteps

Write a key as a spreadsheet

Description

Writes a key as the CSV file that keyFromSpreadsheet() reads, so that a key imported from FishBase or generated from data can be opened in Excel or Google Sheets, edited by hand and read back. The columns are those of the Montana template: Couplet, Choice, Question, Description, ⁠Goes to⁠, Species and Photo, followed by Note and Label when the key's leads have them (a pruned key has notes, see pruneKey(); a PhyloTree key has labels).

Usage

keyToSpreadsheet(key, file, photos = NULL, bom = TRUE)

Arguments

key

A dimoose key.

file

Path of the CSV file to write (UTF-8).

photos

Folder to write the key's embedded photos into; by default the folder the CSV file is in. It is created if needed.

bom

Start the file with a byte-order mark? Excel needs one to show accented and other non-English letters correctly when the file is opened by double-clicking; Google Sheets and keyFromSpreadsheet() read the file either way. Use FALSE for programs that do not expect one.

Details

Photos embedded in the key (see imageMeta()) are written as image files into photos, named by their ids, and the Photo column names those files. A file of that name that is already there with different contents is left alone and used as it is, so writing a key back next to its original photos does not replace them with the smaller embedded copies. Pictures a lead refers to by web address stay web addresses.

Reading the file back gives the same leads and the same photo ids. The exceptions are small: a photo id that cannot be a file name (it has a /, :, comma or the like in it, or differs from another only by capitals) is changed, with a message; a line break inside a cell comes back as a plain line feed whatever it was; and leads without a Choice are lettered.

A spreadsheet holds only those columns. Anything else in the key's leads (for example the taxon links of a FishBase key, or the tests of a key a machine can follow) is not written, and a message says which columns were left out; the title and citation are not written either, so pass desc and meta again to keyFromSpreadsheet().

Value

file, invisibly, so the result can go straight to keyFromSpreadsheet().

See Also

keyFromSpreadsheet()

Examples

dir <- tempfile("key")
dir.create(dir)
csv <- keyToSpreadsheet(montanaKey(), file.path(dir, "montana.csv"))
list.files(dir)
# ... edit the file in a spreadsheet program, then read it back
again <- keyFromSpreadsheet(csv, desc = "Montana trees, edited")
identical(again$leads$Character, montanaKey()$leads$Character)

Leave-one-out evaluation of a key builder

Description

Leave-one-out evaluation of a key builder

Usage

looKey(images, build, score, groups = NULL)

Arguments

images

imageSet() data frame of all images.

build

Function of the training row indices returning a key.

score

Function of (key, test row index) returning a one-row scores matrix for that image.

groups

As in evaluateKey().

Value

As evaluateKey().


Macrohaplogroup of a haplogroup

Description

Macrohaplogroup of a haplogroup

Usage

macroHaplogroup(
  haplogroup,
  level = c("letter", "depth"),
  depth = 2,
  tree = dimoose::phylotree17
)

Arguments

haplogroup

Haplogroup names.

level

"letter" (default) uses the conventional label: L0 to L6 for the African lineages and the leading letter otherwise (H2a is H, U5b is U). This is a naming convention, not a clade: for example D sits inside M phylogenetically. "depth" uses the ancestor at a fixed depth from the root, which is a true clade.

depth

For level = "depth", the depth of the ancestor to report; haplogroups shallower than that are returned as they are.

tree

Default phylotree17.

Value

A character vector the length of haplogroup.

Examples

macroHaplogroup(c("H2a2a1", "L3e1", "U5b"))
macroHaplogroup("H2a", level = "depth", depth = 3)

Graft one key onto a result of another

Description

Joins two keys end to end: the lead of key that ended in the taxon at now leads to the first couplet of subkey. Use it to put a key to the genera of one family under a key to families, or a regional key under a general one.

Usage

mergeKeys(key, subkey, at, labels = c("renumber", "prefix"), prefix = NULL)

Arguments

key

A dimoose key.

subkey

The dimoose key to graft onto it.

at

The result of key that subkey continues from: its whole label, the scientific name in its parentheses or the common name before them (case does not matter). It is an error if no result, or more than one, has that name; a shared result is named in full ("A / B").

labels

How to keep couplet names unique. "renumber" numbers the couplets of the merged key 1, 2, 3, ... in reading order, so the subkey's couplets follow the lead they hang from. "prefix" keeps key's couplet names and puts prefix in front of the subkey's.

prefix

Text put in front of the subkey's couplet names with labels = "prefix"; by default the scientific name of at and a full stop ("Salicaceae.1").

Details

The leads of the two keys are put in one table (a column that only one of them has is left blank for the other), and so are their metadata, embedded images, taxa and features. key's title is kept; subkey's citation is added to key's. If at is the result of several leads, they all lead to the subkey. An embedded image or a feature that the two keys both have under one id, with different contents, is kept apart by giving the subkey's a new id (⁠_2⁠ is added), with a warning for a feature.

Value

A dimoose key.

See Also

pruneKey()

Examples

families <- keyFromLeads(data.frame(
  Statement = c("1", "1"), Choice = c("a", "b"),
  Character = c("Fruit an acorn", "Fruit a capsule releasing cottony seeds"),
  Next = "-", Taxon = c("Fagaceae", "Salicaceae")
), "Two families of trees")
salicaceae <- keyFromLeads(data.frame(
  Statement = c("1", "1"), Choice = c("a", "b"),
  Character = c("Buds with one scale", "Buds with several scales"),
  Next = "-", Taxon = c("Salix", "Populus")
), "Genera of Salicaceae")
both <- mergeKeys(families, salicaceae, at = "Salicaceae")
both$leads
mergeKeys(families, salicaceae, at = "Salicaceae", labels = "prefix")$leads$Statement

Model details as metadata rows

Description

Model details as metadata rows

Usage

modelMeta(model)

Arguments

model

A visionModel().

Value

A key/value data frame with ⁠model_*⁠ keys.


Key to the native broadleaf trees of Montana

Description

A dichotomous key to the eleven native broadleaf (hardwood) trees of Montana, as a table of leads (see keyFromLeads()); montanaKey() turns it into a dimoose key with photos. Written for dimoose: native status was checked in the Montana Field Guide and USDA PLANTS, and every couplet's characters come from Montana Field Guide or Virginia Tech Dendrology descriptions. Common hackberry is left out because the Montana Field Guide gives its origin in Montana as undetermined.

Usage

montana

Format

A data frame with 20 rows (10 couplets) and 7 character columns:

Statement

Couplet number.

Choice

a or b.

Character

Text of the lead.

Next

Couplet the lead leads to, or "-".

Taxon

For terminal leads, "Common name (Genus species)".

Question

What the couplet asks, e.g. "Leaf type?".

Image

Ids of the photos in ⁠inst/extdata/montana/⁠ shown with the lead, separated by ⁠;⁠.

Source

Montana Field Guide, Montana Natural Heritage Program and Montana Fish, Wildlife & Parks, https://fieldguide.mt.gov; USDA PLANTS Database, https://plants.usda.gov; Virginia Tech Dendrology fact sheets, https://dendro.cnre.vt.edu. The page for each species and the source of each couplet are listed in data-raw/montana/sources.md in the package sources.


How common each taxon of a key is, from GBIF occurrence records

Description

Counts the occurrence records GBIF holds for each result of a key, optionally within a country or region. The counts can be given to keyQuality() or summary() as weights, to see how many steps the key takes for the taxa people actually meet.

Usage

occurrenceWeights(
  key,
  region = NULL,
  source = "gbif",
  fetch = qualGbifFetch,
  kingdom = NULL
)

Arguments

key

A dimoose key.

region

Where to count: NULL for the whole world, a two-letter ISO country code ("US"), or a GADM id for a country, state or county ("USA.27_1" is Montana). Anything else is an error. GBIF answers 0 for a GADM id it does not have, so when no taxon has a record in the region a warning asks to check it.

source

Where the counts come from; only "gbif" for now.

fetch

The function that asks GBIF: called as fetch(url, query) with a named list of query parameters, it returns the parsed JSON answer as a list. Replace it to use cached answers or to test without a connection.

kingdom

Optional kingdom ("Plantae", "Animalia", ...), which helps GBIF tell apart names that are used in more than one kingdom.

Details

A result written as "Common name (Genus species)" is looked up by the name in parentheses; other results by the result itself when it looks like a scientific name, which includes genera and families ("Salix", "Salix sp." and "Salix spp." all count the genus) and names followed by their author ("Puma concolor (Linnaeus, 1771)"). A result that lists several taxa the key does not separate, written "A / B" as keyFromRpart() and keyFromMatrix() write them, gets the sum of their counts. Each name is matched to the GBIF Backbone Taxonomy, and the records of the name GBIF accepts for it are counted, so a synonym gets the count of its accepted name. This makes two requests to GBIF per name, one name at a time; after three failed requests in a row the rest are not tried.

A name that GBIF knows in more than one kingdom ("Morus" is a mulberry and a gannet) is not counted unless kingdom says which is meant.

Occurrence counts measure how often a taxon is recorded, which follows how conspicuous and how popular with observers it is as well as how common it is. A taxon with no records gets a weight of 0 and so does not count towards the expected steps at all.

Value

A numeric vector of record counts named by the key's results (as in its Taxon column), NA for results that have no scientific name, that GBIF does not know or cannot tell apart, or whose request failed (each with a warning).

See Also

keyQuality(), checkNames()

Examples

# canned answers in place of GBIF, to show the shape of `fetch`
counts <- c("Acer negundo" = 61000, "Fraxinus pennsylvanica" = 48000)
canned <- function(url, query) {
  if (grepl("species/match", url)) {
    list(usageKey = match(query$name, names(counts)), matchType = "EXACT")
  } else {
    list(count = unname(counts[query$taxonKey]))
  }
}
leads <- data.frame(
  Statement = "1", Choice = c("a", "b"), Character = c("Leaflets 3.", "Leaflets 5 to 7."),
  Next = "-", Taxon = c("Box elder (Acer negundo)", "Green ash (Fraxinus pennsylvanica)")
)
occurrenceWeights(keyFromLeads(leads), fetch = canned)

k <- montanaKey()
w <- occurrenceWeights(k, region = "USA.27_1", kingdom = "Plantae")
w
summary(k, weights = w)

Folder where dimoose keeps downloaded and converted ONNX models

Description

Models are large (tens to hundreds of megabytes), so they are kept outside your project, in R's per-user cache folder for dimoose. Set the option dimoose.onnx.cache to use another folder.

Usage

onnxCache()

Value

A path. The folder is not created until a model is stored in it.

Examples

onnxCache()

Load a vision model that runs on ONNX Runtime, without PyTorch

Description

The light way to embed images and texts: the Python side needs only onnxruntime, numpy, pillow and tokenizers (about 150 MB installed, against 0.8 GB to several GB for PyTorch, depending on the platform). The result is used exactly like a visionModel(): with embedImages(), embedTexts(), scoreImages(), suggestSpecies() and the rest. visionModel(backend = "onnx") calls this function.

Usage

onnxModel(
  name = "clip-vit-base-patch32",
  precision = c("fp32", "fp16", "int8"),
  textPrecision = precision,
  text = TRUE,
  cache = onnxCache(),
  python = NULL,
  threads = 0,
  fetch = NULL
)

Arguments

name

The model: a folder, a Hugging Face repository or a known name (see below).

precision

"fp32", "fp16" or "int8": which copy of the image encoder to use.

textPrecision

The same for the text encoder; by default as precision.

text

If FALSE, the text encoder is not downloaded (about 250 MB less at "fp32"). embedTexts() then fails; sniglet keys do not need it.

cache

Folder for downloaded models.

python

Optional path to the Python executable to use (passed to reticulate::use_python()).

threads

Number of CPU threads for ONNX Runtime; 0 lets it choose.

fetch

For tests: a function ⁠(url, dest)⁠ that downloads a file and returns TRUE, or FALSE if there is no such file.

Value

An object of class dimooseVisionModel, as visionModel() returns, with a files entry: the model's folder (dir), its image and text encoder files (image, text) and, for a model from the Hub, its repository (repo) and the web address of the image encoder (imageUrl).

Where models come from

name is one of

  • a folder on your computer that holds an ONNX model (also written "local-dir:<folder>"), in either layout described below;

  • a Hugging Face repository, "<organisation>/<model>" or "hf-hub:<organisation>/<model>". The files that are needed are downloaded once into onnxCache() and reused; nothing is downloaded when they are already there;

  • "clip-vit-base-patch32" (the default; CLIP ViT-B/32, the same weights as visionModel()) or "clip-vit-base-patch16": the ready-made exports Xenova/clip-vit-base-patch32 and Xenova/clip-vit-base-patch16 on the Hub;

  • "mobileclip-s0": Apple's MobileCLIP-S0 (Xenova/mobileclip_s0), a small model for phones. Its image encoder is 46 MB at "fp32" and 23 MB at "fp16". Its "int8" file is refused: its embeddings are wrong (measured: cosine 0.09 on average with the "fp32" ones);

  • "bioclip", "bioclip-2" or any other model you have converted with exportOnnxModel(). There is no official ONNX copy of BioCLIP on the Hub, so it has to be converted once on a computer with PyTorch; the converted folder can then be copied to computers without it.

Precision and size

Each encoder comes in up to three precisions. Measured on dimoose's 55 example photos against the PyTorch model (CLIP ViT-B/32 image encoder):

precision image encoder cosine with PyTorch (lowest)
"fp32" 352 MB 1.0000
"fp16" 176 MB 1.0000
"int8", converted by exportOnnxModel() 96 MB 0.998
"int8", Xenova/clip-vit-base-patch32 (⁠_quantized⁠) 89 MB 0.81

"fp16" halves the download but runs up to three times slower on a CPU than "fp32". BioCLIP converted by exportOnnxModel() is 345, 173 and 89 MB, with cosines of 1.0000, 1.0000 and 0.998 (BioCLIP 2, a larger model, is about 1217, 609 and 310 MB); on the Montana example photos all three precisions name the same species. A key must be followed with the model and precision it was built with when the precision is "int8": its embeddings differ slightly from the others'. The ready-made "int8" file of Xenova/clip-vit-base-patch32 is a much rougher copy (0.89 on average), so onnxModel() warns when it is used; to get the close one, convert the model yourself with exportOnnxModel("ViT-B-32"). A model you have converted is used in place of the ready-made copy of the same name.

The patch tokens of an "int8" encoder (embedImages(tiles = 0)) are less faithful than its image embeddings: with BioCLIP the lowest cosine of a token with the "fp32" one was 0.86. Use tiles, or a higher precision, to find sniglets with patch tokens.

Layouts of a model folder

dimoose's own layout, written by exportOnnxModel(): dimoose_onnx.json (architecture, image size, preprocessing constants, file names), image_encoder.onnx, text_encoder.onnx, their ⁠_fp16⁠ and ⁠_int8⁠ copies, and tokenizer.json. Its image encoder also returns the model's patch tokens, so embedImages(tiles = 0) works.

The transformers.js layout of ready-made exports on the Hub: config.json, preprocessor_config.json, tokenizer.json, ⁠onnx/vision_model[_fp16|_quantized].onnx⁠ and ⁠onnx/text_model[_fp16|_quantized].onnx⁠. These give one embedding per image and no patch tokens: tiles (the default of embedImages()) work, tiles = 0 does not.

When something goes wrong

  • Could not download ...: the model is fetched from huggingface.co the first time only. Files that arrived are kept, so calling again continues where it stopped. A computer that cannot reach the Hub can be given the model's folder, copied from onnxCache() on one that can, as name.

  • ... is not a readable ONNX model: a model file is damaged, usually a download or copy that stopped early. Delete the file named in the message and load the model again; a file from the Hub is fetched afresh.

  • Could not load dimoose's ONNX code: the Python in use lacks onnxruntime or tokenizers. The message gives the reticulate::py_install() call that installs them.

See Also

exportOnnxModel() to convert a model, exportBrowserModel() to use one in a web page, saveEmbeddings() to need no Python at all.

Examples

## Not run: 
model <- onnxModel()                              # CLIP ViT-B/32, downloaded once
model <- onnxModel(precision = "fp16")            # half the download
images <- imageSet(system.file("extdata", "pics", package = "dimoose"))
emb <- embedImages(model, images)

exportOnnxModel("bioclip")                        # once, needs PyTorch
model <- onnxModel("bioclip")                     # afterwards, without it

## End(Not run)

Build a dimoose object from FishBase key pages

Description

The offline half of importFishbase(): parses a FishBase key description page and its questions (couplet) page, both already downloaded.

Usage

parseFishbase(
  description,
  questions,
  separateTerms = TRUE,
  usePhyloService = c("GNA", "GNR", "none", "TNRS"),
  embedImages = FALSE,
  pageUrl = "https://www.fishbase.se/keys/questions.php"
)

Arguments

description

The description page (keys/description.php): an xml_document, a file path, or a string of HTML.

questions

The questions page (keys/questions.php), in any of the same forms.

separateTerms

If TRUE, split each lead's character statement on ";" into one row per term.

usePhyloService

Taxonomic name resolution: "GNA" verifies terminal taxa with the Global Names verifier (taxize::gna_verifier(), requires the 'taxize' package); "none" skips resolution. "GNR" is accepted as an alias for "GNA". "TNRS" is no longer available because taxize::tnrs() is defunct.

embedImages

If TRUE, download the key's morphology images and store them base64-encoded in meta. Image URLs are always stored.

pageUrl

URL of the questions page, used to resolve relative image links.

Value

A dimoose object.

Examples

## Not run: 
parseFishbase("description.html", "questions.html", usePhyloService = "none")

## End(Not run)

Parse a key written as plain text into a table of leads

Description

The lower-level half of keyFromText(): reads the text and returns one row per lead together with everything that looked wrong, without building a key. Use it to see what the parser made of a difficult text, fix the text or the table, and then call keyFromLeads().

Usage

parseTextKey(
  text,
  llm = NULL,
  strict = FALSE,
  layout = c("auto", "bracketed", "indented", "unnumbered"),
  skip = NULL
)

Arguments

text

The key: a file path, one string holding the whole key, or a character vector of lines. Give the key alone, without the descriptions before and after it.

llm

Optional function for a first pass over messy text; see the section below.

strict

If TRUE, stop when anything had to be repaired (see Details of keyFromText()) instead of repairing it. Problems that cannot be repaired are still returned in problems.

layout

"auto" works out the layout from the text. "indented" always lets a lead without a destination lead to the couplet printed under it; "bracketed" never does. "unnumbered" reads the couplets from the indentation alone (or, without indentation, from the order of the leads) and takes no number at a line start for a couplet number.

skip

Optional regular expression; lines that match are left out (for running heads the parser does not recognise by itself).

Value

A list of class textKeyParse with

  • leads: data frame with one row per lead: Statement (couplet), Choice, Character (the text), Next (next couplet, or "-"), Taxon, Authority, CommonName, Page and Line (the line the lead starts on). Ready for keyFromLeads().

  • problems: data frame with Line, Couplet, Lead, Kind ("problem", "repair" or "note"), Type and Message.

  • title: the first line above the first lead, or NA.

  • layout: "bracketed", "indented", "paired" or "unnumbered".

  • text: the lines that were parsed (the model's answer when llm returned text).

When llm is used, Line is the line of the source text each lead was found on (NA if it could not be found) and leads$LlmLine is the line or row of the model's answer.

See Also

keyFromText()

Examples

# Primes for the second lead, tabs and "go to"
p <- parseTextKey(c(
  "1.  Shell coiled\tgo to 2",
  "1'. Shell of two valves\tBivalvia",
  "2.  Shell with a lid\tLittorina, p. 40",
  "2'. Shell without a lid\tLymnaea"
))
p$leads

# A scanned page: what was repaired and what was skipped
scan <- system.file("extdata", "textkeys", "meadow-ocr.txt", package = "dimoose")
p <- parseTextKey(scan)
p
p$problems[p$problems$Kind == "repair", c("Line", "Message")]

# An indented key without couplet numbers, as in older floras
willows <- "
Capsule glabrous.
    Leaves with close serrations, glaucous beneath.  11. S. melanopsis.
    Leaves entire or with distant serrations.
        Styles none.  12. S. exigua.
        Styles evident; aments dense.  16. S. sessilifolia.
Capsule densely pubescent.  17. S. argophylla.
"
parseTextKey(willows)$leads[, c("Statement", "Choice", "Character", "Next", "Taxon")]

# A stand-in for a language model that returns a table
fake <- function(text) {
  data.frame(
    couplet = c(1, 1), lead = c("a", "b"),
    text = c("Wings present", "Wings absent"),
    destination = c("Pterygota", "Apterygota")
  )
}
parseTextKey("1 Wings present ... Pterygota\n1 Wings absent ... Apterygota", llm = fake)$leads

Add example crops to a sniglet key

Description

Cuts out the example regions of each sniglet and of each lead, stores them as PNGs in the key's features and meta, and points the leads at them so exportWizard() shows them. A "Has" lead shows the sniglet in the images that take that lead; a "Lacks" lead shows what its images have instead (see keyFromSniglets()). The glossary shows each sniglet's examples across all the images.

Usage

patchExemplars(model, key, images, pad = NULL, size = 96)

Arguments

model

A visionModel().

key

A key from keyFromSniglets().

images

The imageSet() the key was built from (for file paths).

pad

Pixels of context to show around the region. By default none for tiles, and half a patch's width for patch tokens.

size

The crop's output size, px.

Value

The key, modified in place and returned invisibly.


PhyloTree Build 17: the human mitochondrial DNA phylogeny

Description

The 5,435 haplogroups of PhyloTree Build 17 (van Oven 2015) as distributed by Haplogrep 3 (release 17.2 of phylotree-rsrs-17), one row per haplogroup, rooted at mtMRCA. Use phylotreeKey() to turn the tree, or part of it, into an identification key, and phylotree17_mutations for the mutations one per row.

Usage

phylotree17

Format

A data frame with 5,435 rows and 5 columns:

haplogroup

Haplogroup name.

parent

Name of the parent haplogroup (NA for mtMRCA).

depth

Number of branches from mtMRCA.

n_subclades

Number of direct subclades.

mutations

Space-separated mutations of the branch leading to this haplogroup, as written by Haplogrep.

Details

Rows are in the order of Haplogrep's tree.xml, which is preorder (each haplogroup directly before its subclades). Names and mutations are Haplogrep's, verbatim: names such as ⁠H2a+152 16311⁠ or K1b1+(16093) label branches that PhyloTree leaves unnamed, and two names (⁠M4’’67⁠, ⁠M4’’67+16311!⁠) contain the Unicode character U+2019. Haplogrep's trailing space on one name is trimmed.

Mutations are in RSRS orientation: each is the state on the branch leading to the haplogroup. ⁠16362C⁠ is a substitution (position and derived base), ⁠263G!⁠ a back-mutation (it reverts the latest change at that position; ⁠!!⁠ marks a second reversion), ⁠315.1C⁠ or ⁠5899.XC⁠ an insertion (X: unspecified length) and ⁠249d⁠ a deletion. Unlike some other copies (e.g. MToolBox's), PhyloTree's unstable mutations, which PhyloTree prints in parentheses, are included; the parentheses are not recorded.

Source

Haplogrep 3 tree phylotree-rsrs-17, release 17.2, commit ⁠0e79ddf1b78f7a1b81fb5d607bc31a6ddea7b18b⁠, src/tree.xml (https://github.com/genepi/phylotree-rsrs-17), MIT license. The conversion and a script that checks these data against GitHub are in ⁠data-raw/phylotree17/⁠ in the package sources.

References

van Oven M (2015). PhyloTree Build 17: Growing the human mitochondrial DNA tree. Forensic Science International: Genetics Supplement Series 5, e392-e394. doi:10.1016/j.fsigss.2015.09.155

Schönherr S, Weissensteiner H, Kronenberg F, Forer L (2023). Haplogrep 3 - an interactive haplogroup classification and analysis platform. Nucleic Acids Research 51, W263-W268. doi:10.1093/nar/gkad284

See Also

phylotree17_mutations, phylotreeKey()

Examples

head(phylotree17)
phylotree17[phylotree17$parent %in% "H2a2a", ]

PhyloTree Build 17 mutations, one per row

Description

The mutations of every haplogroup in phylotree17, one row per mutation, in the order Haplogrep lists them, with the bases before and after worked out by following the tree from the RSRS.

Usage

phylotree17_mutations

Format

A data frame with 13,384 rows and 6 columns:

haplogroup

Haplogroup whose branch carries the mutation.

notation

The mutation as written by Haplogrep.

position

Position in the RSRS/rCRS coordinates.

type

transition, transversion, back-mutation, insertion or deletion.

ancestral

Base before the mutation (for a reverted insertion, the inserted sequence); NA for insertions.

derived

Base after the mutation, or the inserted sequence; NA for deletions and reverted insertions.

Details

Seven back-mutations revert a position that nothing earlier on their path changes (six ⁠16233C!⁠, e.g. in D4c1b1, and D5a2 ⁠1438A!⁠); for those the written base is taken as the result.

Source

See phylotree17.

Examples

table(phylotree17_mutations$type)
phylotree17_mutations[phylotree17_mutations$haplogroup == "J1c", ]

Build an identification key from PhyloTree Build 17

Description

Turns phylotree17, or the part of it below root, into a dimoose key. Each haplogroup with subclades becomes a step at which the user picks the subclade whose defining mutations the sample carries. With paragroups = TRUE every step also offers "none of these", which identifies the paragroup, written as in PhyloTree with a trailing * (e.g. ⁠H2a*⁠).

Usage

phylotreeKey(root = "mtMRCA", paragroups = TRUE, maxDepth = Inf)

Arguments

root

Haplogroup to start from; the default, "mtMRCA", is the whole tree.

paragroups

If TRUE, add a "none of these" choice to every step.

maxDepth

Maximum number of steps below root. Haplogroups at that depth become results even if they have subclades.

Details

The whole tree gives a very large key (2,420 steps and 5,435 possible results with paragroups), so for exportWizard() a subtree or a maxDepth is usually more practical.

Value

A dimoose object whose leads have a Label column with the haplogroup names.

See Also

phylotree17, exportWizard()

Examples

h2a <- phylotreeKey("H2a")
head(h2a$leads)
## Not run: 
exportWizard(phylotreeKey("U5"), "u5.html")
exportWizard(phylotreeKey(maxDepth = 6), "macrohaplogroups.html")

## End(Not run)

Prune a key to some of its taxa

Description

Keeps only the taxa you name (or removes the ones you name) and tidies up what is left: leads that no longer reach any kept taxon are removed, and a couplet that no longer splits anything is taken out, with the lead that led to it now leading straight to where its one remaining lead went. Collapses are repeated until every couplet again has at least two leads going to different places, so chains of couplets, couplets reached from several leads and the first couplet of the key are all handled.

Usage

pruneKey(
  key,
  keep = NULL,
  drop = NULL,
  ranks = c("genus", "exact"),
  note = c("column", "append", "none"),
  renumber = FALSE
)

Arguments

key

A dimoose key.

keep

Taxa to keep; all others are removed. NULL keeps all.

drop

Taxa to remove (after keep has been applied).

ranks

"genus" or "exact"; see 'Matching taxa'.

note

"column", "append" or "none"; see 'What happens to the removed couplet's character'.

renumber

If TRUE, number the remaining couplets 1, 2, 3, ... in reading order (each couplet followed by the couplets its leads go to, first lead first). If FALSE, couplets keep their names.

Details

The leads that remain keep their text, their order and (unless renumber = TRUE) their couplet names, so every kept taxon is reached by the same leads as before, less the ones that no longer decide anything. Pruning in two steps gives the same key as pruning once to the taxa kept by both.

Value

A dimoose key with the same columns in its leads as key (plus Note if a note was written). Embedded images, taxa and features that no remaining lead uses are left out. It is an error if no result is left, or only one (a key needs two results to tell apart).

Matching taxa

Names are compared without regard to case or extra spaces, and the multiplication sign of a hybrid may be typed as a lower-case x ("Populus x acuminata"). An entry of keep or drop matches a taxon of the key when it equals the taxon's whole label ("Bur oak (Quercus macrocarpa)"), the scientific name in its parentheses ("Quercus macrocarpa") or the common name before them ("Bur oak"); entries may themselves be written with or without a common name.

With ranks = "genus" (the default) a name also stands for what lies below it, judged by the words the names begin with:

  • a genus in keep or drop ("Populus") matches every species of that genus in the key, and a species ("Populus balsamifera") matches its subspecies and varieties ("Populus balsamifera subsp. trichocarpa");

  • a name matches the same name followed by its author ("Quercus macrocarpa" and "Quercus macrocarpa Michx.", or "Carcharodon carcharias (Linnaeus, 1758)"), whichever of the two has the author;

  • a species in keep ("Quercus macrocarpa") keeps a result of the key that is only its genus ("Quercus"), and a subspecies keeps its species. This is not done for drop: removing one species does not remove its genus.

With ranks = "exact" only whole labels, scientific names and common names match. Families and higher ranks cannot be told from a name, so a result such as "Salicaceae" is matched only by itself, and a synonym is a different name; regionalKey() asks GBIF or iNaturalist instead and so works at any rank and through synonyms.

Shared results

A result that names several taxa the key cannot tell apart, written "A / B" as keyFromRpart() and keyFromMatrix() write them, is treated as the taxa A and B: each can be kept or dropped by its own name, and when only A is kept the result becomes "A". Naming the whole result ("A / B") keeps or drops all of it. (A label with a common name, such as "Aspen / Popple (Populus tremuloides)", is one taxon.)

What happens to the removed couplet's character

When a couplet is taken out, the text of its one remaining lead still describes the taxa beyond it, and is useful for confirming an identification. With note = "column" (the default) it is added to a Note column on the lead that now skips the couplet; with note = "append" it is added to that lead's Character in square brackets, so it shows wherever the key is displayed; with note = "none" it is discarded. Whatever note is, the removed lead's pictures (Image and ImageLink) are added to those of the lead that now skips the couplet. When the first couplet of the key is taken out there is no lead to carry the text: it goes in the key's metadata as pruned_note (added to any that is already there), and its pictures are left out.

See Also

regionalKey() to prune to the taxa recorded in a place, mergeKeys() to go the other way.

Examples

cottonwoods <- pruneKey(montanaKey(), keep = "Populus")
cottonwoods
cottonwoods$leads[, c("Statement", "Choice", "Next", "Taxon")]

# names can be given with or without the common name
k <- pruneKey(montanaKey(), drop = c("Acer negundo", "Bur oak (Quercus macrocarpa)"),
              renumber = TRUE)
k$validate()
k$leads$Note

Raw FishBase key tables

Description

An R6 object holding a FishBase key's description and its couplet table as downloaded, before conversion to a dimoose object.

Active bindings

desc

Key title (read-only).

df

The key's couplet table: one row per lead, with columns Statement, Choice, Character, Next, Prev and Taxon.

Methods

Public methods


rawhtml$print()

Print the description, table and metadata.

Usage
rawhtml$print(...)
Arguments
...

Unused.


rawhtml$new()

Download a FishBase key.

Usage
rawhtml$new(
  keycode,
  fishbaseUrl = "https://www.fishbase.se/",
  separateTerms = TRUE
)
Arguments
keycode

FishBase key code.

fishbaseUrl

Base URL of the FishBase mirror.

separateTerms

Ignored; kept for backward compatibility.


rawhtml$clone()

The objects of this class are cloneable with this method.

Usage
rawhtml$clone(deep = FALSE)
Arguments
deep

Whether to make a deep clone.


Read and write DELTA datasets

Description

DELTA (DEscription Language for TAxonomy, Dallwitz 1980) is the oldest and still widely used text format for descriptive data, and a TDWG standard. A dataset is classically three files: specs (how many characters there are and of which type), chars (the character list with the states) and items (the taxa, each with its attributes). readDELTA() reads them into a character matrix (see readSDD() for the convention), writeDELTA() writes a matrix as the three files.

Usage

readDELTA(
  specs,
  chars = NULL,
  items = NULL,
  taxon = "Taxon",
  sep = "/",
  inapplicable = NA,
  comments = FALSE
)

writeDELTA(
  x,
  dir,
  characters = NULL,
  taxon = "Taxon",
  sep = "/",
  inapplicable = NULL,
  title = NULL,
  files = c(specs = "specs", chars = "chars", items = "items")
)

Arguments

specs

Path of the specs file, or of a folder holding files named specs, chars and items (any extension, any case).

chars, items

Paths of the chars and items files. May be left out when specs is a folder, or a single file holding everything.

taxon

Name of the taxon column of the matrix.

sep

Separator between the states of a variable taxon in one cell.

inapplicable

For readers, the cell value to give "not applicable" (default NA, the same as unknown). For writers, the cell value that means "not applicable" (default NULL: none).

comments

If TRUE, keep the comments in the names of characters, states and taxa ("pronotum <colour>").

x

A character matrix (data frame), or the list returned by readDELTA(), readSDD(), readLucid() or readXper().

dir

Folder to write the three files into; created if needed.

characters

Optional data frame describing the characters (Character, Type, States, Unit, InapplicableIf); see writeSDD().

title

Title put in the ⁠*SHOW⁠ line of each file.

files

Names of the three files.

Value

readDELTA() returns a tdwgData list like readSDD(), with matrix, characters (Character, Type, States, Unit, InapplicableIf, Number), taxa (Taxon, Comment) and meta. writeDELTA() returns the paths of the three files, invisibly.

What is read

The directives ⁠*CHARACTER LIST⁠, ⁠*ITEM DESCRIPTIONS⁠, ⁠*CHARACTER TYPES⁠, ⁠*NUMBERS OF STATES⁠, ⁠*NUMBER OF CHARACTERS⁠, ⁠*IMPLICIT VALUES⁠ and ⁠*DEPENDENT CHARACTERS⁠, wherever in the given files they are (so a dataset kept in one file, or split differently, reads too); other directives are skipped. Character types are unordered multistate (UM, read as "categorical"), ordered multistate (OM, "ordered"), integer (IN, "integer"), real (RN, "real") and text (TE, "text").

Attributes follow the DELTA standard: ⁠12,1/2⁠ (state 1 or 2) and ⁠12,1&2⁠ (1 and 2) both give "state 1/state 2", so the difference between "or" and "and" is not kept; ⁠13,2-3⁠ is a range, which for an ordered character includes the states in between; ⁠14,U⁠ is unknown (NA), ⁠15,-⁠ not applicable (inapplicable), ⁠16,V⁠ variable (all states). Numeric values are kept as written, e.g. "7-8.5" or "(1-)2-3(-4)", which is the numeric notation of the matrix convention. For a text character the cell is the text of the comment (⁠7<possibly two species>⁠). Implicit values are filled in. A dependent character that is not coded is "not applicable" when its controlling character only takes the controlling states; the rule itself is kept in the InapplicableIf column of characters. Variant items (⁠#+⁠) take the attributes they leave out from their main item, as the standard says.

Comments (text in angle brackets) are dropped from the names of characters, states and taxa unless comments = TRUE, and from the attributes of non-text characters; the comment of a taxon name (usually the authority) is kept in taxa$Comment. A name that consists of nothing but a comment (⁠#4. <longevity of plants>/⁠) is that comment. Typesetting marks (the RTF control words DELTA programs use, such as ⁠\i{}⁠, ⁠\b0{}⁠ or ⁠\par{}⁠) are removed, and those that stand for a character (⁠\endash{}⁠, ⁠\ldblquote{}⁠) become that character. Files that are not valid UTF-8 are read as Windows-1252. The obsolete line sequence numbers are not supported.

The reader was checked against the datasets distributed with Open DELTA (among them the "Grass Genera" sample, a flora of 506 taxa and a virus database of 969 taxa) by comparing every coded attribute with a second, independent parse of the files.

What is written

specs gets the numbers of characters, states and items, the character types, the numbers of states and the dependent characters; chars the character list; items one item per taxon. Every coded cell is written out (no implicit values are used), NA cells are left out, which DELTA reads as unknown, and cells equal to inapplicable are written as -. The states of a variable taxon are joined with / ("or"). A numeric cell that is not a DELTA number or range, and all cells of text characters, are written as comments. Encoding is UTF-8, and lines are wrapped at 78 columns wherever they have a blank to break at.

A few things cannot be written as they are, because DELTA would read them as syntax: angle brackets in a name make a comment (which readDELTA() drops unless comments = TRUE); a star or, in a name, a numero after a blank is put in parentheses (⁠(*)⁠, ⁠(#)⁠); the blank after a slash in a name is dropped. No file written by dimoose has been read into a DELTA program (Confor, Intkey, Open DELTA).

References

Dallwitz, M. J. & Paine, T. A. (1992 onwards). Definition of the DELTA format. https://www.delta-intkey.com/www/standard.htm

See Also

readSDD(), readLucid().

Examples

dir <- system.file("extdata", "tdwg", "beetles-delta", package = "dimoose")
d <- readDELTA(dir)
d
d$matrix
d$characters[, c("Character", "Type", "States", "Unit")]

out <- writeDELTA(d, tempfile("delta"))
out
identical(readDELTA(dirname(out[1]))$matrix, d$matrix)

Read a Haplogrep / PhyloTree tree XML

Description

Parses the nested ⁠<haplogroup>⁠ XML that Haplogrep distributes (for example phylotree-rcrs-17's tree.xml) into the same data frame shape as phylotree17, so any Haplogrep tree can be loaded, not just the bundled snapshot. Each haplogroup's ⁠<details><poly>⁠ entries become its mutations.

Usage

readHaplogrep(file, key = FALSE)

Arguments

file

Path to the tree XML.

key

If TRUE, return a dimoose key built with keyFromLeads() instead of the data frame.

Value

A data frame with haplogroup, parent, depth, n_subclades and mutations, or a dimoose object if key = TRUE.

See Also

phylotree17, phylotreeKey()

Examples

## Not run: 
tree <- readHaplogrep("tree.xml")

## End(Not run)

Read and write Lucid matrix keys

Description

Lucid (Identic / Lucidcentral) is a widely used commercial program for interactive matrix keys: "entities" (taxa) scored for the states of "features" (characters). readLucid() reads a key exported from the Lucid Builder into a character matrix (see readSDD() for the convention) and writeLucid() writes a matrix in a form the Builder imports.

Usage

readLucid(
  file,
  format = NULL,
  taxon = "Taxon",
  sep = "/",
  inapplicable = NA,
  entities = c("columns", "rows"),
  scores = c(1, 2, 3),
  fullNames = FALSE
)

writeLucid(
  x,
  file,
  format = c("csv", "sdd"),
  characters = NULL,
  taxon = "Taxon",
  sep = "/",
  inapplicable = NULL,
  entities = c("columns", "rows"),
  title = NULL
)

Arguments

file

Path of the file to read or write.

format

"csv", "sdd" or (reading only) "lif3"; by default worked out from the file.

taxon

Name of the taxon column of the matrix.

sep

Separator between the states of a variable taxon in one cell.

inapplicable

For readers, the cell value to give "not applicable" (default NA, the same as unknown). For writers, the cell value that means "not applicable" (default NULL: none).

entities

For CSV: whether the entities (taxa) are the "columns" (Lucid's default) or the "rows" (its "transposed" export).

scores

Which Lucid scores mean that the taxon has the state.

fullNames

If TRUE, name characters and taxa by their whole path in Lucid's feature and entity trees ("Flowers:Petals:Colour") rather than by the last step ("Colour"). Names that would be ambiguous are always given in full.

x

A character matrix (data frame), or the list returned by readLucid(), readSDD(), readDELTA() or readXper().

characters

Optional data frame describing the characters (Character, Type, States, Unit); see writeSDD().

title

Title of the dataset (SDD only).

Value

readLucid() returns a tdwgData list like readSDD(): matrix, characters (Character, Type, States, Unit, Path), taxa (Taxon, Path) and meta. writeLucid() returns file, invisibly.

Formats

Lucid has several export formats, and they are not equally well documented:

  • "csv" (Lucid Builder: File > Export Key and Import Key, file type CSV). Documented in the Lucid Builder help, which also offers a small example key with its CSV export: the first row holds the entities, every further row one state named Feature:State (a colon-separated path through the feature tree), and the cells hold the scores 0 absent, 1 present, 2 rarely present, 3 uncertain, 4 present by misinterpretation, 5 rarely present by misinterpretation, 6 not scoped. A numeric feature is one row named Feature# with cells ⁠score type:rare low:low:high:rare high⁠, or a plain 0 when it is not scored. This is the format writeLucid() writes by default, and the documented way to get a matrix into Lucid. Use entities = "rows" for the Builder's "transposed" variant.

  • "sdd": Lucid exports matrix keys as SDD, the TDWG standard; the file is read with readSDD() and written with writeSDD().

  • "lif3" (Lucid Interchange Format 3, .lif3): an XML file. Lucid publishes no schema or description of it beyond saying that it is XML, so readLucid() reads it as far as real exported files show how: features, states and entities (⁠<item>⁠), the feature and entity trees, and the multistate scores. This was checked against four published keys (Lucid 3 and Lucid 4 exports), one of which was also published as SDD: the two readings agree in all 3454 scores. Those files only use the scores 1 (present) and 3 (uncertain); the scores 2, 4, 5 and 6 are taken to mean what they mean in CSV, as they do in the Builder's own key files, which store scores in the same elements. No real LIF3 file with numeric scores was found: they are read when they are laid out as in the Builder's own key files (⁠<scored_data>⁠ with omin, nmin, nmax, omax), and otherwise left out with a warning. Units, dependencies, text scores, subsets and media of a LIF3 file are not read (a warning says so when the file has dependencies or text scores), and dimoose does not write LIF3; export CSV or SDD from the Builder when you need those.

  • LIF (Lucid 2, .lif, a sectioned text file) and the Builder's "XML Dataset" are not publicly specified; readLucid() stops and asks for a CSV or SDD export instead of guessing.

Scores

By default a taxon gets a state when it is scored present (1), rarely present (2) or uncertain (3) for it; states scored only "by misinterpretation" are left out, because the taxon does not really have them. A feature for which every state of a taxon is uncertain, or nothing is scored, gives NA; "not scoped" gives inapplicable. writeLucid() writes 1 for every state in a cell and 0 otherwise, and 6 for cells equal to inapplicable.

For numeric features the Lucid help gives the layout of the cell but not the values its score type can take. The one real export at hand uses 1 for a scored range and a plain 0 for a feature that is not scored, and that is all writeLucid() writes: ⁠1:rare low:low:high:rare high⁠ for a value or range, 0 for NA. A numeric cell equal to inapplicable is also written as 0, with a warning, because no score for "not scoped" is documented for numeric features. On reading, type 1 gives the range, 0 gives NA, 6 is taken as "not scoped" by analogy with the multistate scores, and any other type gives NA with a warning.

Text characters cannot be stored in a Lucid matrix and are left out with a warning. Colons in names are replaced, because Lucid reads them as steps of a path; data that was read from Lucid keeps its paths (the Path columns of characters and taxa). Units of numeric features are not part of the CSV format. No file written by dimoose has been imported into the Lucid Builder.

References

Lucid Builder help, "CSV (Comma Separated Value) Files" and "Importing and Exporting", https://help.lucidcentral.org/lucid/.

See Also

readLucidKey() for Lucid pathway (dichotomous) keys, readSDD(), readDELTA().

Examples

f <- system.file("extdata", "tdwg", "bears-lucid.csv", package = "dimoose")
d <- readLucid(f)
d$matrix
d$characters

out <- writeLucid(d, tempfile(fileext = ".csv"))
readLines(out)
identical(readLucid(out)$matrix, d$matrix)

Read and write Lucid pathway (dichotomous) keys

Description

The Lucid Builder also makes "pathway" keys: ordinary dichotomous keys of couplets and leads (the successor of the stand-alone Lucid Phoenix). readLucidKey() reads one of its exports into a dimoose key and writeLucidKey() writes a dimoose key in the same forms. All four are described in the Lucid Builder help ("Pathway key exports"):

Usage

readLucidKey(file, format = NULL, desc = NULL)

writeLucidKey(key, file, format = c("lpxk", "csv", "text", "sdd"))

Arguments

file

Path of the file to read or write.

format

"lpxk", "csv", "text" or "sdd"; when reading, by default worked out from the file.

desc

Title of the key; by default the title in the file, if any.

key

A dimoose key.

Details

  • "lpxk", Lucid Phoenix XML: ⁠<PhoenixKey>⁠ with the taxa as ⁠<Identity>⁠ elements and the couplets as ⁠<Step>⁠ elements holding ⁠<Lead>⁠s whose goto names the next step or the taxon. The step's text becomes the couplet's Question, an identity's url the TaxonUrl. Filters (⁠<FilterItem>⁠) and icons are not read or written.

  • "csv": one line per lead, ⁠couplet,"lead text",next couplet or taxon⁠.

  • "text": the same with blanks instead of commas, as printed in a paper.

  • "sdd": SDD with an identification key; see readSDDKey().

In the CSV and text forms the last field is read as a couplet when it matches a couplet number and as a taxon otherwise, so writeLucidKey() refuses to write a key in which a taxon is named like a couplet. These two forms hold neither the title of the key nor questions, links or images. The CSV, text and Phoenix XML forms follow the examples printed in the Lucid help; no file exported by the Lucid Builder itself was available to check them against, and no file written by dimoose has been imported into the Builder.

Value

readLucidKey() returns a dimoose key; writeLucidKey() returns file, invisibly.

References

Lucid Builder help, "Pathway key exports", https://help.lucidcentral.org/lucid/pathway-key-exports/.

See Also

readLucid() for Lucid matrix keys, readSDDKey(), keyFromSpreadsheet().

Examples

f <- system.file("extdata", "tdwg", "insects.lpxk", package = "dimoose")
k <- readLucidKey(f)
k
k$leads[, c("Statement", "Choice", "Character", "Next", "Taxon")]

out <- writeLucidKey(k, tempfile(fileext = ".csv"), format = "csv")
readLines(out)

Read and write SDD (TDWG Structured Descriptive Data)

Description

SDD is the XML standard of TDWG (Biodiversity Information Standards) for descriptive data: characters and their states, the taxa, and how each taxon is coded for each character. readSDD() turns an SDD file into a character matrix, writeSDD() writes one. Both follow SDD 1.1 (the ⁠http://rs.tdwg.org/UBIF/2006/⁠ schema), which is what Lucid, Xper2/Xper3 and DELTA tools export.

Usage

readSDD(file, taxon = "Taxon", sep = "/", inapplicable = NA, dataset = NULL)

writeSDD(
  x,
  file,
  characters = NULL,
  taxon = "Taxon",
  title = NULL,
  sep = "/",
  inapplicable = NULL,
  lang = "en",
  key = NULL
)

Arguments

file

Path of the SDD file to read or write.

taxon

Name of the taxon column of the matrix.

sep

Separator between the states of a variable taxon in one cell.

inapplicable

For readers, the cell value to give "not applicable" (default NA, the same as unknown). For writers, the cell value that means "not applicable" (default NULL: none).

dataset

Which ⁠<Dataset>⁠ of the file to read, by position; by default the first one that has coded descriptions (or, for readSDDKey(), identification keys).

x

A character matrix (data frame), or the list returned by readSDD(), readDELTA(), readLucid() or readXper().

characters

Optional data frame describing the characters: a Character column matching the matrix's column names and any of Type, States (to fix the order of the states or list states no taxon has), Unit, InapplicableIf and Cost. By default taken from x if x is a reader's result; otherwise numeric columns are written as numeric characters and all others as categorical ones.

title

Title of the dataset; by default the title in the metadata of x.

lang

Language of the labels, as an xml:lang code.

key

Optional dimoose key to store in the same file as an SDD identification key (see writeSDDKey()).

Value

readSDD() returns a list of class tdwgData with

  • matrix: the character matrix;

  • characters: one row per character with Character (the column name), Type ("categorical", "ordered", "integer", "real" or "text"; SDD itself only tells categorical, real and text apart), States (all defined states, separated by sep), Unit, InapplicableIf ("Other character=state/state": the character does not apply when the other character has one of these states; several rules are separated by ⁠;⁠), Cost when the file has ratings, and Id;

  • taxa: Taxon, Detail and Id;

  • meta: key/value data frame (title, description, language, authors, copyright, generator, created, and keys, the number of identification keys in the file, see readSDDKey()).

writeSDD() returns file, invisibly.

The character matrix

A plain data frame with one row per taxon, the taxon names in a column (Taxon by default) and one column per character. Cells are text:

  • a state name, or several separated by sep when the taxon is variable ("red/yellow");

  • for numeric characters a value or range written the way DELTA and floras write it: "5", "2.3-7.9", "2.3-5.1-7.9" (the middle value is the mean or median) and, with rare extremes, "(1-)2-8(-12)";

  • for text characters the text;

  • NA when the character is unknown, not coded or not applicable (give inapplicable to tell "not applicable" apart).

What is read from SDD

Categorical, quantitative and text characters, in the order of the ⁠<Characters>⁠ list; the coded descriptions (⁠<SummaryData>⁠) of each taxon; ⁠<Status>⁠ codes (NotApplicable becomes inapplicable, the others NA); the dependency rules of the character tree (InapplicableIf, OnlyApplicableIf); and observation-convenience ratings, which become a Cost from 1 (easy) to 5. A taxon with several descriptions gets one row, with the states of all of them. States scored with a "misinterpretation" modifier (Lucid's way of catching a common mistake) are not taken as states of the taxon, and states scored with the modifier the Lucid Builder labels "unscoped" make the cell "not applicable" (the Builder defines that modifier in every SDD file it writes; none of the real exports dimoose was checked against uses it, so how it is attached to a score is assumed to be the same as for the other modifiers). Other modifiers (rarely, uncertain) are ignored and the state kept, so a character whose states are all "uncertain" for a taxon lists all of them. Dependency rules are read from the character itself and from the groups of the character tree it sits in. Quantitative measures map to the range text as follows: Min/Max are the ends of the range, unless UMethLower/UMethUpper are given too, in which case those are the normal range and Min/Max the extremes in parentheses; Mean (or Median, Mode) is the middle value. Other statistics (SD, N) are dropped.

Not read: sample data (⁠<SampleData>⁠ of individual specimens), media, natural-language descriptions, taxon hierarchies, and text in more than one language (the first label of each object is used).

What is written

A complete, schema-valid SDD 1.1 document with the taxon names, the characters, a character tree holding the dependency rules, and one coded description per taxon. NA cells are left out (SDD reads a missing coding as unknown), cells equal to inapplicable get the status NotApplicable. A Cost from 1 to 5 is written as an observation-convenience rating, as Xper3 does. "ordered" characters are written as categorical ones and "integer" as quantitative ones, so those two types come back as "categorical" and "real". Of a cell listing several numeric ranges ("5/10-15") the overall range is written, and numbers are written as numbers ("0.40" comes back as "0.4"). A categorical character for which no taxon has a state is written as a text character, with a warning, because SDD does not allow a categorical character without states. Files written from the real datasets dimoose was tried on (DELTA, Lucid and Xper exports of up to 969 taxa) all validate against the SDD 1.1 schema; none has been imported into Lucid or Xper.

References

Hagedorn, G., Thiele, K., Morris, R. & Heidorn, P. B. (2005). Structured Descriptive Data (SDD) w3c-xml-schema, version 1.0. https://www.tdwg.org/standards/sdd/

See Also

readSDDKey() for dichotomous keys stored in SDD, readXper(), readLucid(), readDELTA().

Examples

f <- system.file("extdata", "tdwg", "violets.sdd.xml", package = "dimoose")
d <- readSDD(f)
d
d$matrix
d$characters[, c("Character", "Type", "States", "Unit")]

# write a matrix of your own
m <- data.frame(
  Taxon = c("Apis mellifera", "Bombus terrestris", "Vespula vulgaris"),
  Hair = c("sparse", "dense", "sparse/absent"),
  "Body length" = c("12-15", "(11-)14-17(-22)", NA),
  check.names = FALSE
)
chars <- data.frame(Character = "Body length", Type = "real", Unit = "mm")
out <- writeSDD(m, tempfile(fileext = ".xml"), characters = chars, title = "Three wasps and bees")
readSDD(out)$matrix

Read and write a dichotomous key stored in SDD

Description

Besides character matrices, an SDD file can hold ready-made identification keys (⁠<IdentificationKey>⁠ elements): a tree of leads, each with a statement and either further leads, a taxon or a pointer to another key in the file. readSDDKey() reads one into a dimoose key, writeSDDKey() stores a dimoose key that way. Lucid's pathway-key builder exports this form.

Usage

readSDDKey(file, key = 1, desc = NULL, dataset = NULL)

writeSDDKey(key, file, title = NULL, lang = "en")

Arguments

file

Path of the SDD file.

key

For readSDDKey(), which key of the file to read: its position or its id. For writeSDDKey(), the dimoose key to write.

desc

Title of the key; by default the key's label in the file.

dataset

Which ⁠<Dataset>⁠ of the file to read, by position; by default the first one that holds a key.

title

Title of the SDD dataset; by default the key's description.

lang

Language of the text, as an xml:lang code.

Details

In SDD the leads form a tree (each lead names its parent lead) and are not numbered. On reading, the leads that share a parent become a couplet and the couplets are numbered in the usual order of a printed key (the first lead of couplet 1 goes to couplet 2, and so on). A lead that points to another key of the same file (⁠<Subkey>⁠) continues with that key's first couplet, so a key split over several SDD keys comes back as one. A question on a lead (or on the key itself) becomes the Question of the couplet it leads to. A lead that ends without a taxon is kept, and reported by key$validate().

On writing, a couplet that is reached from more than one lead (a reticulate key) is written out once under each of them, because an SDD lead can have only one parent. Images and links attached to leads or taxa (Image, TaxonUrl) and other extra columns of the key are not written.

Value

readSDDKey() returns a dimoose key; writeSDDKey() returns file, invisibly.

See Also

readSDD() for the character matrix of an SDD file, readLucidKey() for Lucid's own pathway-key formats.

Examples

f <- system.file("extdata", "tdwg", "insects-key.sdd.xml", package = "dimoose")
k <- readSDDKey(f)
k
k$leads[, c("Statement", "Choice", "Character", "Next", "Taxon")]

out <- writeSDDKey(sharkKey(), tempfile(fileext = ".xml"))
readSDDKey(out)

Read and write Xper2 / Xper3 knowledge bases

Description

Xper3 (and its desktop predecessor Xper2) keep a knowledge base of taxa ("items") described by characters ("descriptors"). Their documented interchange format is SDD: export the knowledge base as SDD from Xper, and import an SDD file written by writeXper() there. Xper's own storage format is not publicly specified, so dimoose does not try to read or write it; these two functions read and write the SDD dialect that Xper produces.

Usage

readXper(file, taxon = "Taxon", sep = "/", inapplicable = NA, strict = TRUE)

writeXper(
  x,
  file,
  characters = NULL,
  taxon = "Taxon",
  title = NULL,
  sep = "/",
  inapplicable = NULL,
  lang = "en"
)

Arguments

file

Path of the SDD file to read or write.

taxon

Name of the taxon column of the matrix.

sep

Separator between the states of a variable taxon in one cell.

inapplicable

For readers, the cell value to give "not applicable" (default NA, the same as unknown). For writers, the cell value that means "not applicable" (default NULL: none).

strict

If TRUE, stop when the file does not say it was generated by Xper.

x

A character matrix (data frame), or the list returned by readSDD(), readDELTA(), readLucid() or readXper().

characters

Optional data frame describing the characters: a Character column matching the matrix's column names and any of Type, States (to fix the order of the states or list states no taxon has), Unit, InapplicableIf and Cost. By default taken from x if x is a reader's result; otherwise numeric columns are written as numeric characters and all others as categorical ones.

title

Title of the dataset; by default the title in the metadata of x.

lang

Language of the labels, as an xml:lang code.

Details

Compared with plain readSDD(), the Xper dialect has a few habits that are handled here: Xper2 names each taxon in the label of its coded description rather than in a taxon list; the dependencies between characters are InapplicableIf rules in a single character tree; and every coding carries an observation-convenience rating, which is read as the character's Cost (1 = easy to 5 = hard). readXper() stops if the file was not written by Xper, unless strict = FALSE.

Value

readXper() returns the same tdwgData list as readSDD(); writeXper() returns file, invisibly.

References

Vignes-Lebbe, R., Chesselet, P. & Diep Thi, M.-H. (2016). Xper3: new tools for collaborating, training and transmitting knowledge on botanical phenotypes. In: Botanists of the twenty-first century, UNESCO. https://www.xper3.fr/

See Also

readSDD(), writeSDD().

Examples

f <- system.file("extdata", "tdwg", "felids-xper3.sdd.xml", package = "dimoose")
d <- readXper(f)
d$matrix
d$characters[, c("Character", "InapplicableIf", "Cost")]

out <- writeXper(d, tempfile(fileext = ".sdd.xml"))
identical(readXper(out)$matrix, d$matrix)

Recurrent (homoplasic) mutation sites

Description

Sites that mutate independently on many branches of the tree, such as 152, 16311 and 16189 in PhyloTree. These are the variants whose effects a tree can find across unrelated lineages.

Usage

recurrentPositions(
  min_branches = 10,
  types = c("transition", "transversion", "back-mutation"),
  mutations = dimoose::phylotree17_mutations
)

Arguments

min_branches

Minimum number of branches (haplogroups) on which the site mutates.

types

Mutation types to count; by default substitutions and back-mutations. Insertions and deletions are left out, except that a reverted insertion counts as a back-mutation at its insertion site (for example "5899.1").

mutations

Default phylotree17_mutations.

Value

A data frame of site and n_branches, most recurrent first.

Examples

head(recurrentPositions())

A key to the taxa recorded in a place

Description

Asks GBIF or iNaturalist which of the key's taxa have been recorded in a place and prunes the key to those with pruneKey(): the key to the trees of Flathead County from a key to the trees of Montana, in one line.

Usage

regionalKey(
  key,
  place,
  source = c("gbif", "inat"),
  minRecords = 1,
  kingdom = NULL,
  unknown = c("keep", "drop"),
  filters = list(),
  ...,
  fetch = prnFetch
)

Arguments

key

A dimoose key.

place

The place; see the section for the source used.

source

"gbif" or "inat".

minRecords

Keep a taxon with at least this many records.

kingdom

For GBIF, the kingdom of the key's taxa ("Plantae", "Animalia", ...), which tells apart names used in more than one. For iNaturalist, the name or numeric id of any taxon that contains all the key's taxa ("Plantae", "Aves").

unknown

What to do with results the source does not know by name (such as "Other"): "keep" them, with a warning, or "drop" them.

filters

Named list of further search parameters for the count: for GBIF, occurrence search parameters such as list(basisOfRecord = "HUMAN_OBSERVATION", year = "2000,2025"); for iNaturalist, observation search parameters such as list(quality_grade = "any").

...

Passed to pruneKey(): note and renumber.

fetch

Function that makes a request: called as fetch(url, query) with query a named list, it returns the parsed JSON as nested lists. Replace it to cache or log requests, or to test without a network. The default asks a busy or unreachable server up to three times.

Details

Each result of the key is looked up by the scientific name in its parentheses ("Bur oak (Quercus macrocarpa)"), or by the whole result when there is none; the taxa of a shared result ("A / B") are looked up one by one, and the result keeps those that are recorded. The key's names need not be the source's: a synonym is counted under the name the source accepts for it, and the result keeps the key's own label. Results at any rank work: a genus or a family counts the records of all its species.

The sources are asked one taxon at a time, so a key of 70 taxa takes about 150 requests of GBIF (about half a minute). If a request fails, it is an error that says which taxon was being asked about, and nothing is pruned.

Value

A dimoose key, pruned. Its attribute "records" (attr(x, "records")) is a data frame with a row for every taxon of the original key: Taxon, name (the name looked up), matched (the name the source counted it under), id (the source's id for that), records (NA when the source does not know the name) and kept. The attribute "place" says which place was used. When every taxon is kept the key is returned unpruned. It is an error if the source knows none of the key's taxa by name, or if fewer than two taxa are left.

GBIF

Each name is matched to the GBIF Backbone (⁠/v1/species/match⁠) and its occurrences in the place are counted (⁠/v1/occurrence/search⁠ with limit=0), two requests per taxon. A synonym is counted under its accepted name, and a species that GBIF files under one of its subspecies is counted as the species. A name GBIF has in more than one kingdom is settled by kingdom, or, for a genus, by taking the accepted genus. Only records with occurrenceStatus=PRESENT are counted. place can be

  • a GADM id, such as "USA.27.15_1" for Flathead County, Montana, or "USA.27_1" for Montana (gadmGid);

  • a two-letter country code, such as "CH" (country);

  • a polygon in WKT, "POLYGON((...))", with its points anticlockwise (geometry);

  • a named list of GBIF occurrence search parameters, used as they are, such as list(country = "US", stateProvince = "Montana");

  • the name of a region, looked up in GADM (⁠/v1/geocode/gadm/search⁠). Add the regions it lies in after commas to tell places of the same name apart: "Flathead County, Montana", "Flathead County, MT", "Montana, USA". A name that still fits several regions, or none, is an error that lists the regions GADM has with their ids; give the id of the one you mean. The message printed at the end names the region used.

iNaturalist

The place is found with ⁠/v1/places/autocomplete⁠ (or give its numeric id). Parts after commas pick among places of one name ("Lake County, MT"). When several places match, the one of exactly that name is used (if there are several, a country or state before a county or a park, then the largest), and a message names it and lists the others with their ids, so that you can give the id of another. The species observed in the place are listed with ⁠/v1/observations/species_counts⁠, 500 a page, counting research-grade observations only. A result of the key is counted by name when a listed species has that name; otherwise it is looked up with ⁠/v1/taxa⁠ (which also finds synonyms) and counts every listed species below it, or, for a subspecies or variety, its own observations. Give kingdom to list only that group's species, which saves pages. When the list would take more requests than asking about each taxon (a whole country, with no kingdom), a message says so and each taxon's observations in the place are counted with two requests instead. Requests made by the default fetch are a second apart, as iNaturalist asks.

Examples

flathead <- regionalKey(montanaKey(), "USA.27.15_1", kingdom = "Plantae")
flathead
attr(flathead, "records")

regionalKey(montanaKey(), "Flathead County", source = "inat", kingdom = "Plantae")

Rename sniglets

Description

Sniglets start with coined names ("dulmzil") because nobody has looked at them yet. After looking at what a sniglet picks out (the example crops in the wizard's glossary), give it a real name. The coined word stays as the sniglet's permanent identifier, so scores, machine tests and saved name tables keep working; only the text people read changes: the leads ("Has round headlights"), the questions and the glossary.

Usage

renameSniglets(x, names = NULL, definitions = NULL)

Arguments

x

A key from keyFromSniglets(), or a discoverSniglets() result (so that every key built from it afterwards uses the names).

names

The new names: a named character vector (c(dulmzil = "an egg-crate grille")), a names table as described above, or the path of a CSV file holding one.

definitions

Optional named character vector of definitions, named like names. names may be NULL to change definitions only.

Details

The names table has one row per sniglet you want to rename, with columns sniglet (the coined word, or the sniglet's current name), name and optionally definition. Rules:

  • a blank name puts the coined word back;

  • no two sniglets may share a name, and a name may not be another sniglet's coined word;

  • a definition column replaces the definitions of the rows listed (blank removes one); without the column, definitions are left alone;

  • a used column, as snigletNames() returns, is ignored.

Names read best as noun phrases ("an egg-crate grille"), since leads are written "Has name" and "Lacks name".

Value

A renamed copy of x; x itself is not changed.

See Also

snigletNames()

Examples

set.seed(1)
patches <- array(rnorm(5 * 9 * 8), c(5, 9, 8))
patches <- sweep(patches, c(1, 2), sqrt(apply(patches^2, c(1, 2), sum)), "/")
s <- discoverSniglets(patches, k = 3)
first <- s$sniglets$sniglet[1]
s <- renameSniglets(s, stats::setNames("round headlights", first),
                    definitions = stats::setNames("Circular lamps in the grille", first))
snigletNames(s)

Save embeddings to a file, and read them back

Description

Embedding images needs a vision model, and so Python; using embeddings does not. saveEmbeddings() writes what embedImages() or embedTexts() returned to one small file, and loadEmbeddings() reads it back as it was, row names and tiling included, on a computer with no Python at all. Everything dimoose does after embedding (discoverSniglets(), snigletScores(), keyFromSniglets(), keyFromClusters(), featureScores(), ...) works from the loaded embeddings.

Usage

saveEmbeddings(x, file, precision = c("float32", "float16"), model = NULL)

loadEmbeddings(file)

Arguments

x

A numeric matrix or array, or a list of them such as embedImages() returns (entries that are NULL are left out).

file

File to write or read; .safetensors is the usual ending.

precision

"float32", or "float16" for a file half the size. Embeddings have unit length, so 16-bit floats change cosine similarities only in the fourth decimal place.

model

Optional visionModel() the embeddings were made with; its details are stored in the file. Embeddings from loadEmbeddings() keep the model they were loaded with when saved again.

Value

saveEmbeddings() returns file, invisibly. loadEmbeddings() returns what was saved: a matrix or array, or a list of them. If the file records a model, its details are in the result's model attribute.

File format

The file is in the safetensors format, which Python (safetensors.numpy.load_file()), JavaScript and other languages read too:

  • 8 bytes: the length N of the header in bytes, an unsigned 64-bit little-endian integer;

  • N bytes: the header, one JSON object in UTF-8, padded with spaces so that N is a multiple of 8. It has one entry per array, in the order the arrays are stored, named image and patches (for embedImages()), embeddings (for a single matrix) or by the names of the list given. Each entry has dtype ("F32" or "F16"), shape (the dimensions, e.g. ⁠[36, 512]⁠) and data_offsets (⁠[begin, end]⁠: the array's first byte and the byte after its last, counted from the end of the header);

  • the arrays' numbers, one array after another with nothing between them, in row-major (C) order: an ⁠images x d⁠ matrix is stored image by image, an ⁠images x tiles x d⁠ array image by image and tile by tile.

Numbers are IEEE 754 floats of 4 bytes ("F32") or 2 bytes ("F16"), and always little-endian, whatever the computer that writes or reads the file.

The header's ⁠__metadata__⁠ entry is an object of strings: format ("dimoose-embeddings"), version ("1"), single ("true" when x was one array and not a list, so that it is read back as one), ⁠rownames.<array>⁠ (a JSON array of the row names: image ids or texts; other dimensions' names are not kept), ⁠tiles.<array>⁠ and ⁠empty.<array>⁠ (the tiling and the rule for empty tiles, see embedImages(), as numbers separated by commas) and model (a JSON object with the model's details, if there is a model).

loadEmbeddings() also reads safetensors files written by other programs, as long as their arrays are 16-, 32- or 64-bit floats.

Precision

R keeps numbers as 64-bit floats and the file holds 32-bit ones (about 7 significant digits). Vision models compute in 32 bits, so what embedImages() and embedTexts() return is saved and read back exactly; other numbers come back rounded to the nearest 32-bit float. With precision = "float16" every number is rounded to about 3 significant digits: unit-length embeddings change by at most 0.0005 per coordinate. Numbers too large for the chosen precision become Inf, and NA is read back as NaN.

Embeddings that come with dimoose

The embeddings of the package's example photos are in its extdata/embeddings folder, so the examples can be followed with no Python: ⁠pics-clip-vit-b-32.safetensors⁠ (the 36 fruit photos in extdata/pics, whole photos and 4 x 4 tiles), ⁠new-photos-clip-vit-b-32.safetensors⁠ (the 6 photos in extdata/new-photos), ⁠texts-clip-vit-b-32.safetensors⁠ (the prompts of the vision vignette) and montana-bioclip.safetensors (the tree photos in extdata/montana and the names of montanaKey()'s species, embedded with BioCLIP). The folder's README.md says how each was made.

Examples

# stand-ins for embedImages(model, images): 3 images, 2 tiles each, 4 dimensions
emb <- list(image = matrix(rnorm(12), 3, 4, dimnames = list(c("a", "b", "c"), NULL)),
            patches = array(rnorm(24), c(3, 2, 4), dimnames = list(c("a", "b", "c"), NULL, NULL)))
attr(emb$patches, "tiles") <- 1L
file <- tempfile(fileext = ".safetensors")
saveEmbeddings(emb, file)
back <- loadEmbeddings(file)
all.equal(back$image, emb$image, tolerance = 1e-6)
rownames(back$image)
attr(back$patches, "tiles")

# the example photos' embeddings, made with CLIP ViT-B/32: a key with no Python
pics <- loadEmbeddings(system.file("extdata", "embeddings", "pics-clip-vit-b-32.safetensors",
                                   package = "dimoose"))
dim(pics$patches)
attr(pics, "model")$arch
images <- imageSet(system.file("extdata", "pics", package = "dimoose"))
sniglets <- discoverSniglets(pics$patches, k = 40)
scores <- snigletScores(pics$patches, sniglets$centroids)
keyFromSniglets(scores, images, sniglets, method = "rpart")

Score new images against a key's features

Description

Embeds the images (once per distinct crop region the key's text features use) and applies featureScores().

Usage

scoreImages(key, model, paths)

Arguments

key

A key with a features table.

model

A visionModel().

paths

Image files, or an imageSet() (its path column is embedded and its id column names the rows).

Value

An ⁠images x features⁠ matrix for classify().


Names of a key's sniglets

Description

Lists the sniglets of a key (or of a discoverSniglets() result) with the names they currently go by. This is the table renameSniglets() takes: write it to a CSV file, fill in the name and definition columns, and read it back.

Usage

snigletNames(x)

Arguments

x

A key from keyFromSniglets(), or a discoverSniglets() result.

Value

A data frame with one row per sniglet:

  • sniglet: the coined word. It identifies the sniglet and never changes.

  • name: the name people see. It is the coined word until you rename it.

  • definition: an optional sentence saying what the sniglet is.

  • used (keys only): whether the key asks about this sniglet. Only these need names.

See Also

renameSniglets()


Score images against sniglets

Description

Score images against sniglets

Usage

snigletScores(patches, centroids)

termScores(patches, centroids)

Arguments

patches

As in discoverSniglets().

centroids

A ⁠k x d⁠ matrix of sniglet centroids (unit rows), e.g. discoverSniglets(...)$centroids.

Value

An ⁠images x sniglets⁠ matrix: for each sniglet, the highest cosine similarity between the centroid and any patch of the image. Columns are named by the coined words, whatever the sniglets have been renamed to.


Split photos into training and test sets by observation

Description

Photos of one observation show the same individual, often seconds apart. Putting one in the training set and another in the test set would make a key look better than it is, so whole observations go to one side.

Usage

splitByObservation(observation, label, test = 0.3, seed = 1)

Arguments

observation

The observation each photo belongs to. A photo without one (NA) counts as an observation of its own.

label

The class of each photo. The split is made within each class, so every class keeps about the same share for testing.

test

Share of each class's observations to test with.

seed

Seed for the draw.

Value

A character vector, "train" or "test" for each photo. A class with two or more observations gets at least one on each side; a class with a single observation goes to the training set. An observation with photos in several classes stays whole too: all its photos go to the training set, so nothing in the test set has a sibling in training.

Examples

obs <- c(1, 1, 2, 3, 4, 5, 5, 6)
label <- rep(c("oak", "elm"), each = 4)
data.frame(obs, label, split = splitByObservation(obs, label, test = 0.5))

Have a vision-language model suggest names for sniglets

Description

A sniglet starts with a coined word ("dulmzil") because nobody has looked at it yet. This function does the first look for you: it shows a vision-language model each sniglet's example crops (and, for contrast, crops from photos in which the sniglet was not found, when there are any) and asks for a short name and a one-sentence definition. The result is a names table for renameSniglets(), so naming becomes reviewing suggestions instead of writing every name from scratch. Nothing is renamed until you pass the table (or the CSV file) to renameSniglets().

Usage

suggestSnigletNames(
  x,
  model,
  vision = NULL,
  images = NULL,
  scores = NULL,
  crops = NULL,
  sniglets = NULL,
  labels = NULL,
  examples = 4,
  contrast = 4,
  maxWords = 3,
  retries = 2,
  file = NULL,
  pause = 0,
  size = 96,
  resume = FALSE
)

Arguments

x

A key from keyFromSniglets(), or a discoverSniglets() result.

model

The vision-language model: a function ⁠function(images, prompt)⁠ that returns the model's reply as text. images is a list with one entry per crop, each a list with data (the image, base64 encoded), mediaType (e.g. "image/png") and role ("example" or "contrast"); the examples come first. claudeNamer() makes such a function for the Anthropic API.

vision, images

A visionModel() and the imageSet() the sniglets were found in, to cut crops when x does not carry them.

scores

Optional ⁠images x sniglets⁠ matrix from snigletScores(), used to pick contrast crops for a sniglet set.

crops

Optional crops of your own: a list named by coined word, each entry either a character vector of examples or a list with examples and contrast. A crop is an image file, a ⁠data:⁠ URI or a base64 string.

sniglets

Which sniglets to name (coined words or current names). By default the ones a key uses, or all of a sniglet set.

labels

The class names, which a sniglet name must not contain. By default the key's results, or the labels of images.

examples, contrast

How many example and contrast crops to show per sniglet at most.

maxWords

Longest name allowed, in words.

retries

How many times to ask again when a name cannot be used.

file

Optional path of a CSV file to write the table to. It is the spreadsheet renameSniglets() reads: review it, edit the name and definition columns, then renameSniglets(x, file). It is written after every sniglet.

pause

Seconds to wait between sniglets, for APIs with a rate limit.

size

Size in pixels of crops cut with vision.

resume

TRUE to carry on from an existing file instead of starting again; see "Long runs".

Value

A data frame with one row per sniglet, in the shape renameSniglets() takes: sniglet (the coined word), name and definition (the suggestion, or what the sniglet already had when there is no usable suggestion), followed by status ("ok", "suffixed", "rejected", "error" or "no crops") and note. renameSniglets() ignores the last two columns.

Where the crops come from

  • A key that has been through patchExemplars() carries its crops, so nothing else is needed. The contrast crops are those of the key's "Lacks" leads.

  • For a key without crops, or a discoverSniglets() result, give vision and images and the crops are cut for you. For a sniglet set, scores (from snigletScores()) adds contrast crops: the best-matching region of the photos that match the sniglet least.

  • Or give your own in crops.

What makes a usable name

Each suggestion is checked, and the model is asked again (up to retries times, with the reason) when a name

  • has more than maxWords words (a leading "a", "an" or "the" is not counted);

  • contains the name of one of the classes (labels), also in the plural: a sniglet is something you can see on a specimen, such as "a serrate margin", not the species it was found on. For labels written "Common name (Genus species)" the common name, the scientific name and the genus are all refused, and so is the last word of a common name of several words ("aspen" of "Quaking aspen", so "aspen bark" is refused), unless every class ends in it or it is a general word such as "tree" or "flower";

  • is already the name or the coined word of another sniglet. Capitals, punctuation and a leading article make no difference: "Serrate margin" and "a serrate margin" are the same name;

  • is no name at all ("unknown", "n/a", or nothing but punctuation).

A leading "has" or "lacks" is taken off first, since the key adds its own.

If the last answer is still a duplicate it gets a number ("peeling bark 2", status "suffixed"). If it still breaks another rule, or the model fails (an API error, a rate limit, a refusal, a reply that cannot be read), the sniglet keeps the name it had and the row says why in status and note; the other sniglets are not affected.

Long runs

With file, the table is written after every sniglet, so a run that is interrupted or stopped by an error leaves what it had. resume = TRUE picks such a file up: sniglets it already has a usable name for (status "ok" or "suffixed") are not asked about again, the others are. Without resume an existing file is overwritten. The folder of file must exist.

See Also

claudeNamer(), renameSniglets(), snigletNames()

Examples

set.seed(1)
patches <- array(rnorm(5 * 9 * 8), c(5, 9, 8))
patches <- sweep(patches, c(1, 2), sqrt(apply(patches^2, c(1, 2), sum)), "/")
s <- discoverSniglets(patches, k = 2)
pics <- imageSet(system.file("extdata", "pics", package = "dimoose"))
crops <- stats::setNames(list(pics$path[1:2], pics$path[3:4]), s$sniglets$sniglet)
# a stand-in for a real model, which would look at the crops
replies <- c('{"name": "a waxy skin", "definition": "Smooth skin with a dull shine."}',
             '{"name": "a ribbed surface", "definition": "Parallel ridges along the fruit."}')
asked <- 0
fake <- function(images, prompt) { asked <<- asked + 1; replies[asked] }
suggestions <- suggestSnigletNames(s, fake, crops = crops, labels = unique(pics$label))
suggestions
snigletNames(renameSniglets(s, suggestions))

Suggest which of a key's species a photo shows

Description

Compares each photo with the names of the key's results and ranks the results by how well they match, so a photo can be checked against the species the key covers. This works best with a model trained on names of organisms, such as BioCLIP 2 (visionModel(name = "bioclip-2")): it recognises a species from its scientific name without being shown any examples. Only the key's own species are considered, which makes the answers much more reliable than asking the model to name a species out of all those it knows.

Usage

suggestSpecies(key, model, paths, top = 3, prompt = "a photo of %s.")

Arguments

key

A dimoose key.

model

A visionModel().

paths

Image files, or an imageSet() (its path column is embedded and its id column names the rows).

top

How many of the best-matching results to return per photo.

prompt

The text each name is put into, with ⁠%s⁠ where the name goes. The default is the one BioCLIP was trained with.

Details

Each result is described by its scientific name when it is written as "Common name (Genus species)", and by the whole result otherwise. The probabilities are a softmax over all the key's results with the temperature CLIP models are trained with (cosine similarity times 100).

Value

A data frame with top rows per image: image (its id), rank, Taxon, probability and similarity (cosine similarity).

See Also

identifyPhotos() for one answer per photo, at the genus or family when the species is uncertain.

Examples

## Not run: 
model <- visionModel(name = "bioclip-2")
suggestSpecies(montanaKey(), model, "my-photo.jpg")

## End(Not run)

Export a key as Auspice (Nextstrain) JSON v2

Description

Writes the key's spanning tree as an Auspice v2 dataset, which the Nextstrain viewer and auspice.us can display. Node names are made unique and space-free (required by the schema); the original labels and each lead's text are kept in node_attrs. div is the depth from the root.

Usage

toAuspiceJSON(key, title = NULL, file = NULL)

Arguments

key

A dimoose key.

title

Dataset title; defaults to the key's description.

file

Optional path to write to; if NULL, the JSON is returned as a string.

Value

The JSON string (invisibly if written to file).

See Also

toNewick(), toTreedata()

Examples

js <- toAuspiceJSON(phylotreeKey("H2a"))

Convert a key to an igraph directed graph

Description

Unlike toNewick() and toDataTree(), which use a spanning tree, this keeps the key's true structure: a couplet reached from several leads has several incoming edges. Each edge carries the lead's label and Character text; vertices are couplets and taxa.

Usage

toIgraph(key)

Arguments

key

A dimoose key.

Value

An igraph::igraph directed graph.

Examples

## Not run: 
g <- toIgraph(arachnidaKey())
igraph::vcount(g)

## End(Not run)

Write a key as Newick, with safe labels or node annotations

Description

The dimoose method ⁠$toNewick()⁠ writes quoted Newick, which is correct but which several parsers (ape, scikit-bio, toytree) mangle. This wrapper adds two options for interoperating with the wider phylogenetics ecosystem.

Usage

toNewick(
  key,
  labels = c("quoted", "safe"),
  annotate = c("none", "nhx", "beast")
)

Arguments

key

A dimoose key.

labels

"quoted" (the default, identical to key$toNewick()) or "safe", which replaces every label with an ASCII id (⁠n<couplet>⁠ for couplets, ⁠t<n>⁠ for taxa) and returns the mapping in a side table, so fragile parsers never see spaces or punctuation. The real text travels in the companion table (this is also the Newick-plus-table shape Taxonium reads).

annotate

"none" (default), "nhx" or "beast": attach each lead's Character text to the node it leads to, as an NHX (⁠[&&NHX:lead=...]⁠) or BEAST-style (⁠[&lead=...]⁠) comment. Values are sanitized (⁠, : ; = " [ ]⁠ and newlines removed) because Newick comments cannot carry them. Only one of labels = "safe" or annotate applies at a time; annotate wins if both are given.

[ ]: R:%20

Value

For labels = "quoted" or any annotate, a single Newick string. For labels = "safe" with annotate = "none", a list of newick (the string) and labels (a data frame of id and label).

See Also

toTreedata(), toAuspiceJSON()

Examples

k <- arachnidaKey()
cat(toNewick(k, annotate = "nhx"))

Convert a key to a tidytree treedata (for ggtree)

Description

Builds a treedata object whose tree is the key's spanning tree and whose node data carries each lead's Character, Choice and Taxon. The data of a child node describes the edge into it, so in ggtree you draw lead text on branches with geom_label(aes(x = branch, label = Character)) and join your own data by the node column (never by label, since taxa repeat).

Usage

toTreedata(key)

Arguments

key

A dimoose key.

Value

A tidytree::treedata object.

See Also

toNewick(), toIgraph()

Examples

## Not run: 
td <- toTreedata(arachnidaKey())
library(ggtree)
ggtree(td) + geom_tiplab() + geom_label(aes(x = branch, label = Character))

## End(Not run)

Vibrio consensus identification matrix

Description

Biochemical and growth test results for 74 Vibrio taxa from Noguerola & Blanch (2008), one row per taxon. vibrioKey() generates a key from it and vibrioMatrix() recodes it for keyFromMatrix().

Usage

vibrio

Format

A data frame with 74 rows and 46 columns: species, then 45 tests coded as in the paper: + positive, - negative, ⁠(+)⁠ / ⁠(-)⁠ mostly positive / negative, v or d variable, ND not determined (and one blank cell in the source, NA). Tests under the paper's "Acid from" and "Resistance to" headings are named with that prefix.

Source

Noguerola I, Blanch AR (2008). Identification of Vibrio spp. with a set of dichotomous keys. Journal of Applied Microbiology 105, 175-185. doi:10.1111/j.1365-2672.2008.03730.x. Extracted from the paper's table; see ⁠data-raw/vibrio/⁠ in the package sources.


The vibrio dataset as a character matrix

Description

Recodes the vibrio consensus matrix of Noguerola & Blanch (2008) into the character matrix keyFromMatrix() takes: + and ⁠(+)⁠ (mostly positive) become "+", - and ⁠(-)⁠ become "-", the variable results v and d become "+/-", and ND (not determined) and the one blank cell become NA.

Usage

vibrioMatrix()

Value

A data frame with 74 rows: Taxon, then the 45 tests.

See Also

vibrioKey()

Examples

m <- vibrioMatrix()
m[1:4, 1:4]
table(unlist(m[-1]), useNA = "ifany")

Load a vision model for embedding images

Description

Uses Python's open_clip through reticulate. You do not need to install anything in Python yourself: the first call sets up a private Python with the packages dimoose needs (torch, open_clip_torch, pillow, numpy) and reuses it afterwards. That first call downloads them, which takes a few minutes. This needs reticulate 1.41 or later.

Usage

visionModel(
  weights = NULL,
  name = "ViT-B-32",
  pretrained = "openai",
  python = NULL,
  backend = c("torch", "onnx"),
  precision = c("fp32", "fp16", "int8")
)

Arguments

weights

Local checkpoint file: OpenAI's ⁠ViT-B-32.pt⁠, or an open_clip checkpoint for the architecture name. If NULL, the weights are downloaded.

name

The model: an open_clip architecture name, "bioclip", "bioclip-2", "hf-hub:<organisation>/<model>" or "local-dir:<folder>" (see below).

pretrained

open_clip's tag for the weights to download for an architecture name; not used for the other kinds of name.

python

Optional path to the Python executable to use (passed to reticulate::use_python()).

backend

"torch" (PyTorch and open_clip) or "onnx" (ONNX Runtime, a much smaller install; see below).

precision

For backend = "onnx": "fp32", "fp16" or "int8", see onnxModel().

Value

An object of class dimooseVisionModel with py (the Python model) and spec (library, architecture, weights SHA-256, image size, ...).

A lighter setup, without PyTorch

Most of that first download is PyTorch. visionModel(backend = "onnx") runs the same model on ONNX Runtime instead, which needs only the Python packages onnxruntime, tokenizers, pillow and numpy (about 150 MB installed) and gives the same embeddings (cosine 1.0000 with the PyTorch ones on dimoose's example photos). CLIP ViT-B/32 and ViT-B/16 are downloaded ready-made; BioCLIP and other models are converted once with exportOnnxModel(). See onnxModel(), which this calls, for where models come from and for precision. Use one backend per R session: the Python that reticulate sets up for the first one lacks the other's packages.

To use a Python of your own instead, give its path in python, or select it with reticulate::use_python(), reticulate::use_virtualenv() or the RETICULATE_PYTHON environment variable before the first call, and install the packages there with reticulate::py_install(c("torch", "open_clip_torch", "pillow", "numpy")).

Other models, including BioCLIP

Any open_clip model whose image side is a vision transformer works. The key builders only see its embeddings, so nothing else changes.

  • visionModel(name = "bioclip") loads BioCLIP and visionModel(name = "bioclip-2") BioCLIP 2, models trained on images of organisms, from the Hugging Face Hub. They are downloaded on first use.

  • name = "hf-hub:<organisation>/<model>" loads any other open_clip model on the Hub, and name = "local-dir:<folder>" a copy you have downloaded (the folder holds the model's open_clip_config.json and weights file).

  • name and pretrained together select one of open_clip's built-in models, e.g. ⁠name = "ViT-B-16", pretrained = "laion2b_s34b_b88k"⁠.

A key can only be followed with the model it was built with: use the same model for embedImages() and later for scoreImages().

BioCLIP's text side was trained on names of organisms, not on descriptions of what they look like. It suits sniglets (discoverSniglets()), which use images only, better than vocabulary questions (keyFromClusters()).

Examples

## Not run: 
model <- visionModel()                    # CLIP ViT-B/32
model <- visionModel(name = "bioclip")    # BioCLIP, for photos of organisms
model <- visionModel(backend = "onnx")    # CLIP ViT-B/32 without PyTorch

## End(Not run)

Prompts for a vocabulary of visual features

Description

Prompts for a vocabulary of visual features

Usage

vocabularyPrompts(vocabulary, template = "{x}")

Arguments

vocabulary

Data frame with feature and opposite columns (and an optional region, "y0,y1,x0,x1" fractions).

template

Text with {x} where the feature goes, e.g. "a pickup truck with {x}".

Value

A character vector: the feature prompt then the opposite prompt for each row, in order. Embed these with embedTexts() and pass the result to keyFromClusters().

Examples

v <- data.frame(feature = c("round lights", "a flat hood"),
                opposite = c("square lights", "a curved hood"))
vocabularyPrompts(v, "a truck with {x}")