API Reference

Tags and Map Persistence

The tag is the central organizing piece of data in HTMap. Every map that you run produces a Map which is connected to a unique tag. A tag cannot be re-used until the associated map has been deleted or retagged. You can either provide a tag or let HTMap generate one automatically.

If you do not provide a tag, the map will be marked as transient. Transient maps will be removed by the htmap.clean() function without passing all = True, while non-transient (i.e., persistent) maps will not. If you provide a tag during map creation or htmap.Map.retag() a map, it will be marked as persistent.

Mapping Functions

htmap.map(func, args, map_options=None, tag=None, quiet=False)[source]

Map a function call over a one-dimensional iterable of arguments. The function must take exactly one positional argument and no keyword arguments.

Parameters:
Return type:

Map

Returns:

map – A htmap.Map representing the map.

htmap.starmap(func, args=None, kwargs=None, map_options=None, tag=None, quiet=False)[source]

Map a function call over aligned iterables of arguments and keyword arguments. Each element of args and kwargs is unpacked into the signature of the function, so their elements should be tuples and dictionaries corresponding to position and keyword arguments of the mapped function.

Parameters:
Return type:

Map

Returns:

map – A htmap.Map representing the map.

htmap.build_map(func, map_options=None, tag=None)[source]

Return a MapBuilder for the given function.

Parameters:
Return type:

MapBuilder

Returns:

map_builder – A MapBuilder for the given function.

Map Builder

class htmap.MapBuilder(func, map_options=None, tag=None)[source]

The htmap.MapBuilder provides an alternate way to create maps. Once created via htmap.build_map() or similar as a context manager, the map builder can be called as if it were the function you’re mapping over. When the with block exits, the inputs are collected and submitted as a single map.

with htmap.build_map(tag="pow", func=lambda x, p: x ** p) as builder:
    for x in range(1, 4):
        builder(x, x)

map = builder.map
print(list(map))  # [1, 4, 27]
__call__(*args, **kwargs)[source]

Adds the given inputs to the map.

Return type:

None

__len__()[source]

The length of a MapBuilder is the number of inputs it has been sent.

Return type:

int

property map: Map

The Map associated with this MapBuilder. Will raise htmap.exceptions.NoMapYet when accessed until the with block for this MapBuilder completes.

MappedFunction

A more convenient and flexible way to work with HTMap is to use the htmap() decorator to build a MappedFunction.

htmap.mapped(map_options=None)[source]

A decorator that wraps a function in an MappedFunction, which provides an interface for mapping functions calls out to an HTCondor cluster.

Parameters:

map_options (Optional[MapOptions]) – An instance of htmap.MapOptions. Any map calls from the MappedFunction produced by this decorator will inherit from this.

Return type:

Union[Callable, MappedFunction]

Returns:

mapped_function – A MappedFunction that wraps the function (or a wrapper function that does the wrapping).

class htmap.MappedFunction(func, map_options=None)[source]
Parameters:
map(args, tag=None, map_options=None)[source]

As htmap.map(), but the func argument is the mapped function.

Return type:

Map

starmap(args=None, kwargs=None, tag=None, map_options=None)[source]

As htmap.starmap(), but the func argument is the mapped function.

Return type:

Map

build_map(tag=None, map_options=None)[source]

As htmap.build_map(), but the func argument is the mapped function.

Return type:

MapBuilder

Map

The Map is your window into the status and output of your map. Once you get a map result back from a map call, you can use its methods to get the status of jobs, change the properties of the map while its running, pause, restart, or cancel the map, and finally retrieve the output once the map is done.

The various methods that allow you to get and iterate over components will raise exceptions if something has gone wrong with your map:

The exception message will contain information about what caused the error. See Error Handling for more details on error handling.

class htmap.Map(*, tag, map_dir)[source]

Represents the results from a map call.

Warning

You should never instantiate a Map directly! Instead, you’ll get your Map by calling a top-level mapping function like htmap.map(), a MappedFunction mapping method, or by using htmap.load(). We are not responsible for whatever vile contraption you build if you bypass the correct methods!

__len__()[source]

The length of a Map is the number of components it contains.

__getitem__(item)[source]

Return the output associated with the input index. Does not block.

Return type:

Any

classmethod load(tag)[source]

Load a Map by looking up its tag.

Raises htmap.exceptions.TagNotFound if the tag does not exist.

Parameters:

tag (str) – The tag to search for.

Return type:

Map

Returns:

map – The map with the given tag.

property components: Tuple[int, ...]

Return a tuple containing the component indices for the htmap.Map.

property is_done: bool

True if all of the output is available for this map.

property is_active: bool

True if any map components are not complete (or errored!).

wait(timeout=None, show_progress_bar=False, holds_ok=False, errors_ok=False)[source]

Wait until all output associated with this Map is available.

If any components in the map are held or experience an execution error, this method will raise an exception (htmap.exceptions.MapComponentHeld or htmap.exceptions.MapComponentError, respectively).

Parameters:
  • timeout (Union[int, float, timedelta, None]) – How long to wait for the map to complete before raising a htmap.exceptions.TimeoutError. If None, wait forever.

  • show_progress_bar (bool) – If True, a progress bar will be displayed.

  • holds_ok (bool) – If True, will not raise exceptions if components are held.

  • errors_ok (bool) – If True, will not raise exceptions if components experience execution errors.

Return type:

None

get(component, timeout=None)[source]

Return the output associated with the input component index. If the component experienced an execution error, this will raise htmap.exceptions.MapComponentError. Use get_err(), errors(), error_reports() to see what went wrong!

Parameters:
Return type:

Any

get_err(component, timeout=None)[source]

Return the error associated with the input component index. If the component actually succeeded, this will raise htmap.exceptions.ExpectedError.

Parameters:
Return type:

ComponentError

iter(timeout=None)[source]

Returns an iterator over the output of the htmap.Map in the same order as the inputs, waiting on each individual output to become available.

Parameters:

timeout (Union[int, float, timedelta, None]) – How long to wait for each output to be available before raising a htmap.exceptions.TimeoutError. If None, wait forever.

Return type:

Iterator[Any]

iter_with_inputs(timeout=None)[source]

Returns an iterator over the inputs and output of the htmap.Map in the same order as the inputs, waiting on each individual output to become available.

Parameters:

timeout (Union[int, float, timedelta, None]) – How long to wait for each output to be available before raising a htmap.exceptions.TimeoutError. If None, wait forever.

Return type:

Iterator[Tuple[Tuple[tuple, Dict[str, Any]], Any]]

iter_as_available(timeout=None)[source]

Returns an iterator over the output of the htmap.Map, yielding individual outputs as they become available.

The iteration order is initially random, but is consistent within a single interpreter session once the map is completed.

Parameters:

timeout (Union[int, float, timedelta, None]) – How long to wait for the entire iteration to complete before raising a htmap.exceptions.TimeoutError. If None, wait forever.

Return type:

Iterator[Any]

iter_as_available_with_inputs(timeout=None)[source]

Returns an iterator over the inputs and output of the htmap.Map, yielding individual (input, output) pairs as they become available.

The iteration order is initially random, but is consistent within a single interpreter session once the map is completed.

Parameters:

timeout (Union[int, float, timedelta, None]) – How long to wait for the entire iteration to complete before raising a htmap.exceptions.TimeoutError. If None, wait forever.

Return type:

Iterator[Tuple[Tuple[tuple, Dict[str, Any]], Any]]

iter_inputs()[source]

Returns an iterator over the inputs of the htmap.Map.

Return type:

Iterator[Any]

property component_statuses: List[ComponentStatus]

Return the current state.ComponentStatus of each component in the map.

components_by_status()[source]

Return the component indices grouped by their states.

Return type:

Mapping[ComponentStatus, Tuple[int, ...]]

Examples

This example finds the completed jobs for a submitted map, and processes those results:

from time import sleep
import htmap

def job(x):
    sleep(x)
    return 1 / x


m = htmap.map(job, [0, 2, 4, 6, 8], tag="foo")

# Wait for all jobs to finish.
# Alternatively, use `futures = htmap.load("foo")` on a different process
sleep(10)

completed = m.components_by_status()[htmap.JobStatus.COMPLETED]
for component in completed:
    result = m.get(future)
    # Whatever processing needs to be done
    print(result)  # prints "2", "4", "6", and "8"
status()[source]

Return a string containing the number of jobs in each status.

Return type:

str

property holds: Dict[int, ComponentHold]

A dictionary of component indices to their Hold (if they are held).

hold_report()[source]

Return a string containing a formatted table describing any held components.

Return type:

str

property errors: Dict[int, ComponentError]

A dictionary of component indices to their ExecutionError (if that component experienced an error).

error_reports()[source]

Yields the error reports for any components that experienced an error during execution.

Return type:

Iterator[str]

property memory_usage: List[int]

Return the latest peak memory usage of each map component, measured in MB. A component that hasn’t reported yet will show a 0.

Warning

Due to current limitations in HTCondor, memory use for very short-lived components (<5 seconds) will not be accurate.

property runtime: List[timedelta]

Return the total runtime (user + system) of each component.

property local_data: int

Return the number of bytes stored on the local disk by the map.

remove(force=False)[source]

This command removes a map from the Condor queue. Functionally, this command aborts a job.

This function will completely remove a map from the Condor queue regardless of job state (running, executing, waiting, etc). All data associated with a removed map is permanently deleted.

Parameters:

force (bool) – If True, do not wait for HTCondor to remove the map components before removing local data.

Return type:

None

property exists: bool

True if and only if the map has not been successfully removed. Otherwise, False.

hold()[source]

This command holds a map. The components of the map will not be allowed to run until released (see Map.release()).

HTCondor may itself hold your map components if it detects that something has gone wrong with them. Resolve the underlying problem, then use the Map.release() command to allow the components to run again.

Return type:

None

release()[source]

This command releases a map, undoing holds (see Map.hold()). The held components of a released map will become idle again.

HTCondor may itself hold your map components if it detects that something has gone wrong with them. Resolve the underlying problem, then use this command to allow the components to run again.

Return type:

None

pause()[source]

This command pauses a map. The running components of a paused map will keep their resource claims, but will stop actively executing. The map can be un-paused by resuming it (see the Map.resume() command).

Return type:

None

resume()[source]

This command resumes a map (reverses the Map.pause() command). The running components of a resumed map will resume execution on their claimed resources.

Return type:

None

vacate()[source]

This command vacates a map. The running components of a vacated map will give up their claimed resources and become idle again.

Checkpointing maps will still have access to their last checkpoint, and will resume from it as if execution was interrupted for any other reason.

Return type:

None

set_memory(memory)[source]

Change the amount of memory (RAM) each map component needs.

Warning

Edits do not affect components that are currently running. To “restart” components so that they see the new attribute value, consider vacating their map (see the vacate command).

Parameters:

memory (int) – The amount of memory (RAM) to request, as an integer number of MB.

Return type:

None

set_disk(disk)[source]

Change the amount of disk space each map component needs.

Warning

Edits do not affect components that are currently running. To “restart” components so that they see the new attribute value, consider vacating their map (see the vacate command).

Parameters:

disk (int) – The amount of disk space to request, as an integer number of KB.

Return type:

None

rerun(components=None)[source]

Re-run (part of) the map from scratch. The selected components must be completed or errored.

Any existing output of re-run components is removed; they are re-submitted to the HTCondor queue with their original map options (i.e., without any subsequent edits).

Parameters:

components (Optional[Iterable[int]]) – The components to rerun. If None, the entire map will be re-run.

Return type:

None

retag(tag)[source]

Give this map a new tag. The old tag will be available for re-use immediately.

Retagging a map makes it not transient. Maps that have never had an explicit tag given to them are transient and can be easily cleaned up via the clean command.

Parameters:

tag (str) – The tag to assign to the map.

Return type:

None

property is_transient: bool

True is the map is transient, False otherwise.

property stdout: MapStdOut

A sequence containing the stdout for each map component. You can index into it (with a component index) to get the stdout for that component, or iterate over the sequence to get all of the stdout from the map.

property stderr: MapStdErr

A sequence containing the stderr for each map component. You can index into it (with a component index) to get the stderr for that component, or iterate over the sequence to get all of the stderr from the map.

property output_files: MapOutputFiles

A sequence containing the path to the directory containing the output files for each map component. You can index into it (with a component index) to get the path for that component, or iterate over the sequence to get all of the paths from the map.

count(value) integer -- return number of occurrences of value
index(value[, start[, stop]]) integer -- return first index of value.

Raises ValueError if the value is not present.

Supporting start and stop arguments is optional, but recommended.

class htmap.ComponentStatus(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]

An enumeration of the possible statuses that a map component can be in. These are mostly identical to the HTCondor job statuses of the same name.

UNKNOWN = 'UNKNOWN'
UNMATERIALIZED = 'UNMATERIALIZED'
IDLE = 'IDLE'
RUNNING = 'RUNNING'
REMOVED = 'REMOVED'
COMPLETED = 'COMPLETED'
HELD = 'HELD'
SUSPENDED = 'SUSPENDED'
ERRORED = 'ERRORED'
classmethod display_statuses()[source]
Return type:

Tuple[ComponentStatus, ...]

class htmap.MapStdOut(map)[source]

An object that helps implement a map’s sequence over its stdout. Don’t both instantiating one yourself: use the Map.stdout attribute instead.

get(component, timeout=None)

Return a string containing the stdout/stderr from a single map component.

Parameters:
Return type:

str

Returns:

stdx – The standard output/error of the map component.

class htmap.MapStdErr(map)[source]

An object that helps implement a map’s sequence over its stderr. Don’t both instantiating one yourself: use the Map.stderr attribute instead.

get(component, timeout=None)

Return a string containing the stdout/stderr from a single map component.

Parameters:
Return type:

str

Returns:

stdx – The standard output/error of the map component.

class htmap.MapOutputFiles(map)[source]

An object that helps implement a map’s sequence over its output file directories. Don’t both instantiating one yourself: use the Map.output_files attribute instead.

get(component, timeout=None)[source]

Return the pathlib.Path to the directory containing the output files for the given component.

Parameters:
Return type:

Path

Returns:

path – The path to the directory containing the output files for the given component.

Error Handling

Map components can generally encounter two kinds of errors:

  • An exception occurred inside your function on the execute node.

  • HTCondor was unable to run the map component for some reason.

The first kind will result in HTMap transporting a htmap.ComponentError back to you, which you can access via htmap.Map.get_err(). The htmap.ComponentError.report() method returns a formatted error report for your perusal. htmap.Map.error_reports() is a shortcut that returns all of the error reports for all of the components of your map. If you want to access the error programmatically, you can grab it using htmap.get_err().

The second kind of error doesn’t provide as much information. The method htmap.Map.holds() will give you a dictionary mapping components to their htmap.ComponentHold, if they have one. htmap.Map.hold_report() will return a formatted table showing any holds in your map. The hold’s reason attribute will tell you a lot about what HTCondor doesn’t like about your component.

class htmap.ComponentError(*, map, component, exception_msg, node_info, python_info, scratch_dir_contents, stack_summary)[source]

Represents an error experienced by a map component during remote execution.

map

The htmap.Map the component is a part of.

Type:

htmap.Map

component

The component index from the map.

Type:

int

exception_msg

The raw message string from the remote exception.

Type:

str

node_info

A tuple containing information about the HTCondor execute node the component ran on.

Type:

tuple

python_info

A tuple containing information about the Python installation on the execute node.

Type:

tuple

scratch_dir_contents

A list of paths in the scratch directory on the execute node.

Type:

List[pathlib.Path]

stack_summary

The Python stack frames at the time of execution, excluding HTMap’s own stack frame.

Type:

traceback.StackSummary

report()[source]

Return a formatted error report.

The raw information in this report is available in the attributes of this class.

Return type:

str

class htmap.ComponentHold(code, reason)[source]

Represents an HTCondor hold on a map component.

Parameters:
  • code (int) – The HTCondor HoldReasonCode.

  • reason (str) – The HTCondor hold reason.

MapOptions

Map options are the equivalent of HTCondor’s submit descriptors. All HTCondor submit descriptors are valid map options except those reserved by HTMap for internal use (see below).

Fixed options are the most basic option. The entire map will used the fixed option. If you pass a single string as the value of a map option, it will become a fixed option.

Variadic options are options that are given individually to each component of a map. For example, each component of a map might need a different amount of memory. In that case you could pass a list to request_memory, with the same number of elements as the number of inputs to the map.

Inherited options are given to a htmap.MappedFunction when it is created. Any maps made using that function can inherit these options. Options that are passed in the actual map call override inherited options (excepting fixed_input_files, see the note). For example, if you know that a certain function always takes a large amount of memory, you could give it a large request_memory at the htmap.MappedFunction level so that you don’t have to do it for every individual map. Additionally, default map options can be set globally via settings['MAP_OPTIONS.<option_name>'] = <option_value>.

Warning

Only certain options make sense as inherited options. For example, they shouldn’t be variadic options.

fixed_input_files has special behavior as an inherited option: they are merged together instead of overridden.

Note

When looking at examples of raw HTCondor submit files, you may see submit descriptors that are prefixed with a + or a MY.. Those options should be passed to htmap.MapOptions via the custom_options keyword arguments.

class htmap.MapOptions(*, fixed_input_files=None, input_files=None, output_remaps=None, custom_options=None, **kwargs)[source]
Parameters:
  • fixed_input_files (Union[PathLike, TransferPath, Iterable[Union[PathLike, TransferPath]], None]) – A single file, or an iterable of files, to send to all components of the map.

  • input_files (Union[Iterable[Union[PathLike, TransferPath]], Iterable[Iterable[Union[PathLike, TransferPath]]], None]) – An iterable of single files or iterables of files to map over. This may be useful if you want additional files to be sent to each map component, but don’t want them in your mapped function’s arguments.

  • output_remaps (Union[Mapping[str, TransferPath], Iterable[Mapping[str, TransferPath]], None]) – A dictionary, or an iterable of dictionaries, specifying output transfer remaps. A remapped output file is sent to a specified destination instead of back to the submit machine. If a single dictionary is passed, it will be applied to every map component (in this case, you may want to use the $(component) submit macro to differentiate them). Each dictionary should be a “mapping” between the names (last path component, as a string) of o utput files and their destinations, given as a TransferPath. You must still call transfer_output_files() on the files for the them to be transferred at all; listing them here only sets up the remapping.

  • custom_options (Optional[Dict[str, str]]) – A dictionary of submit descriptors that are not built-in HTCondor descriptors. These are the descriptors that, if you were writing a submit file, would have a leading + or MY.. The leading characters are unnecessary here, but can be included if you’d like.

  • kwargs (Union[str, Iterable[str]]) – Additional keyword arguments are interpreted as HTCondor submit descriptors. Values that are single strings are used for all components of the map. Providing an iterable for the value will map that option. Certain keywords are reserved for internal use (see the RESERVED_KEYS class attribute).

Notes

Warning

The representation of the values in fixed_input_files, input_files, custom_options and kwargs should exactly match the characters in the submit file after the =.

For example, let’s say your job requires this submit file:

# file: job.submit
foo = "bar"
aaa = xyz
bbb = false
ccc = 1

The MapOptions that express the same submit options would be:

>>> options = {"foo": '"bar"', "aaa": "xyz", "bbb": "false", "ccc": "1"}
>>> print(options["foo"])  # exactly matches the value in the submit file
... "bar"
>>> options["foo"] = "\"bar\""  # alternative value
>>> MapOptions(**options)

Submit file values with quotes require escaped quotes in the Python string.

RESERVED_KEYS = {'+IsHTMapJob', '+component', 'IsHTMapJob', 'MY.IsHTMapJob', 'MY.component', 'arguments', 'component', 'executable', 'jobbatchname', 'log', 'should_transfer_files', 'stderr', 'stdout', 'submit_event_notes', 'transfer_executable', 'transfer_input_files', 'transfer_output_files', 'transfer_output_remaps', 'universe', 'when_to_transfer_output'}
classmethod merge(*others)[source]

Merge any number of MapOptions together, like a collections.ChainMap. Options closer to the left take priority. :rtype: MapOptions

Note

fixed_input_files is a special case, and is merged up the chain instead of being overwritten. requirements are also combined, in a way where all requirements must be satisfied.

File Transfer

class htmap.TransferPath(path, protocol=None, location=None)[source]

A TransferPath describes the location of a file or directory. If the protocol and location are both None, it describes a location on the local filesystem. If either are given, it describes a remote location.

When used as an argument to a mapped function, a TransferPath tells HTMap to arrange for the specified files/directories to be transferred to the execute machine from some location, which may be the local filesystem on the submit machine or some remote location like an HTTP address or an S3 server.

Transfer paths are recognized in mapped function inputs as long as they are either:

  1. Arguments or keyword arguments of the mapped function.

  2. Stored inside a primitive container (tuple, list, set, dictionary value) that is an argument or keyword argument of the mapped function. Nested containers are inspected recursively.

When the mapped function runs execute-side, it will receive (instead of this object) a normal pathlib.Path object pointing to the execute-side path of the file/directory.

TransferPath is also used to specify the locations for output files to be sent, if they are not to be returned to the submit machine. For example, output files could be sent to an S3 server. See the output_remaps argument of MapOptions for more details on “remapped” output file transfer.

Where appropriate, TransferPath has the same interface as a pathlib.Path. See the examples for some ways to leverage this API to efficiently construct transfer paths.

Attention

You may need to pass additional submit descriptors to your map to actually be able to use input/output transfers for certain protocols. For example, to transfer to and from an S3 server, you also need to pass aws_access_key_id_file and aws_secret_access_key_file. See the condor_submit documentation for more details.

Examples

Transfer a file stored in your home directory using HTCondor file transfer:

transfer_path = htmap.TransferPath.cwd() / 'file.txt'

Transfer a local file at an absolute path using HTCondor file transfer:

transfer_path = htmap.TransferPath("/foo/bar/baz.txt")

Get a file from an HTTP server, located at http://htmap.readthedocs.io/en/latest/_static/htmap-logo.svg:

transfer_path = htmap.TransferPath(
    path = "en/latest/_static/htmap-logo.svg",
    protocol = "http",
    location = "htmap.readthedocs.io",
)

or

base_path = htmap.TransferPath(
    path = "/",
    protocol = "http",
    location = "htmap.readthedocs.io",
)
transfer_path = base_path / 'en' / 'latest' / '_static' / 'htmap-logo.svg'
Parameters:
  • path (Union[TransferPath, PathLike]) – The path to the file or directory to transfer.

  • protocol (Optional[str]) – The protocol to perform for the transfer with. If set to None (the default), use HTCondor local file transfer.

  • location (Optional[str]) – The location to find a remote file when using a protocol transfer. This could be the address of a server, for example.

htmap.transfer_output_files(*paths)[source]

Informs HTMap about the existence of output files.

Attention

This function is a no-op when executing locally, so you if you’re testing your function it won’t do anything.

Attention

The files will be moved by this function, so they will not be available in their original locations.

Parameters:

paths (PathLike) – The paths to the output files.

Return type:

None

Checkpointing

htmap.checkpoint(*paths)[source]

Informs HTMap about the existence of checkpoint files. This function should be called every time the checkpoint files are changed, even if they have the same names as before.

Attention

This function is a no-op when executing locally (i.e., not execute-side), so you if you’re testing your function locally it won’t do anything.

Attention

The files will be copied by this function, so try not to make the checkpoint files too large.

Parameters:

paths (PathLike) – The paths to the checkpoint files.

Return type:

None

Management

These functions help you manage your maps.

htmap.status(maps=None, include_state=True, include_meta=True)[source]

Return a formatted table containing information on the given maps.

Parameters:
  • maps (Optional[Iterable[Map]]) – The maps to display information on. If None, displays information on all existing maps.

  • include_state (bool) – If True, include information on the state of the map’s components.

  • include_meta (bool) – If True, include information about the map’s memory usage, disk usage, and runtime.

Return type:

str

Returns:

table – A text table containing information on the given maps.

htmap.get_tags(pattern=None)[source]

Return a tuple containing the tag for all existing maps, with optional filtering based on a glob-style pattern.

Parameters:

pattern (Optional[str]) – A glob-style pattern. Only tags that fit the pattern will be returned. If None (the default), all tags will be returned.

Return type:

Tuple[str, ...]

Returns:

tags – A tuple containing the tags that match the pattern.

htmap.load(tag)[source]

Reconstruct a Map from its tag.

Parameters:

tag (str) – The tag to search for.

Return type:

Map

Returns:

map – The result with the given tag.

htmap.load_maps(pattern=None)[source]

Return a tuple containing the Map for all existing maps, with optional filtering based on a glob-style pattern.

Parameters:

pattern (Optional[str]) –

A glob-style pattern. Only maps whose tags fit the pattern will be returned. If None (the default), all maps will be returned.

Return type:

Tuple[Map, ...]

Returns:

maps – A tuple contain the maps whose tags fit the pattern.

htmap.remove(tag, not_exist_ok=True)[source]

Remove the map with the given tag.

Parameters:
  • tag (str) – The tag to search for and remove.

  • not_exist_ok (bool) – If False, raise htmap.exceptions.MapIdNotFound if the tag doesn’t exist.

Return type:

None

htmap.clean(*, all=False)[source]

Clean up transient maps by removing them.

Maps that have never had a tag explicitly set are assigned randomized tags and marked as “transient”. This command removes maps marked transient (and can also remove all maps, not just transient ones, if the –all option is passed).

Parameters:

all (bool) – If True, remove all maps, not just transient ones. Defaults to False.

Return type:

List[str]

Returns:

cleaned_tags – A list of the tags of the maps that were removed.

Programmatic Status Messages

These functions are useful for generating machine-readable status information.

htmap.status_json(maps=None, include_state=True, include_meta=True, compact=False)[source]

Return a JSON-formatted string containing information on the given maps.

Disk and memory usage are reported in bytes. Runtimes are reported in seconds.

Parameters:
  • maps (Optional[Iterable[Map]]) – The maps to display information on. If None, displays information on all existing maps.

  • include_state (bool) – If True, include information on the state of the map’s components.

  • include_meta (bool) – If True, include information about the map’s memory usage, disk usage, and runtime.

  • compact (bool) – If True, the JSON will be formatted in the most compact possible representation.

Return type:

str

Returns:

json – A JSON-formatted dictionary containing information on the given maps.

htmap.status_csv(maps=None, include_state=True, include_meta=True)[source]

Return a CSV-formatted string containing information on the given maps.

Disk and memory usage are reported in bytes. Runtimes are reported in seconds.

Parameters:
  • maps (Optional[Iterable[Map]]) – The maps to display information on. If None, displays information on all existing maps.

  • include_state (bool) – If True, include information on the state of the map’s components.

  • include_meta (bool) – If True, include information about the map’s memory usage, disk usage, and runtime.

Return type:

str

Returns:

csv – A CSV-formatted table containing information on the given maps.

Delivery Methods

htmap.register_delivery_method(name, descriptors_func, setup_func=None)[source]

Register a new delivery method with HTMap.

Parameters:
  • name (str) – The name of the delivery method; this is what the DELIVERY_METHOD should be set to to use this delivery method.

  • descriptors_func (Callable[[str, Path], dict]) – The function that provides the HTCondor submit descriptors for this delivery method.

  • setup_func (Optional[Callable[[str, Path], None]]) – The function that does any setup necessary to running the map.

Return type:

None

Transplant Installs

These functions help you manage your transplant installs.

htmap.transplants()[source]
Return type:

Tuple[Transplant, ...]

class htmap.Transplant(hash: str, path: Path, created: datetime, size: int, packages: Tuple[str, ...])[source]

An object that represents metadata information about a transplant install.

Create new instance of Transplant(hash, path, created, size, packages)

hash: str

Alias for field number 0

path: Path

Alias for field number 1

created: datetime

Alias for field number 2

size: int

Alias for field number 3

packages: Tuple[str, ...]

Alias for field number 4

classmethod load(path)[source]
Parameters:

path (Path) – The path to the transplant install.

Return type:

Transplant

Returns:

transplant – The Transplant that represents the transplant install.

remove()[source]
htmap.transplant_info()[source]
Return type:

str

Settings

HTMap exposes configurable settings through htmap.settings, which is an instance of the class htmap.settings.Settings. This settings object manages a lookup chain of dictionaries. The settings object created during startup contains two dictionaries. The lowest level contains HTMap’s default settings, and the next level up is constructed from a settings file at ~/.htmaprc. If that file does not exist, an empty dictionary is used instead. The file should be formatted in TOML.

Alternate settings could be stored in other files or constructed at runtime. HTMap provides tools for saving, loading, merging, prepending, and appending settings to each other. Each map is search in order, so earlier settings can flexibly override later settings.

Warning

To entirely replace your settings, do not do htmap.settings = <new settings object>. Instead, use the htmap.settings.Settings.replace() method. Replacing the settings by assignment breaks the internal linking between the settings objects and its dependencies.

Hint

These may be helpful when constructing fresh settings:

  • HTMap’s base settings are available as htmap.BASE_SETTINGS.

  • The settings loaded from ~/.htmaprc are available as htmap.USER_SETTINGS.

class htmap.settings.Settings(*settings)[source]
get(key, default=None)[source]
Return type:

Any

to_dict()[source]

Return a single dictionary with all of the settings in this Settings, merged according to the lookup rules.

Return type:

dict

replace(other)[source]

Change the settings of this Settings to be the settings from another Settings.

Return type:

None

append(other)[source]

Add a map to the end of the search (i.e., it will be searched last, and be overridden by anything before it).

Parameters:

other (Union[Settings, dict]) – Another settings-like object to insert into the Settings.

Return type:

None

prepend(other)[source]

Add a map to the beginning of the search (i.e., it will be searched first, and override anything after it).

Parameters:

other (Union[Settings, dict]) – Another settings-like object to insert into the Settings.

Return type:

None

classmethod from_settings(*settings)[source]

Construct a new Settings from another Settings.

Return type:

Settings

classmethod load(path)[source]

Load a Settings from a file at the given path.

Return type:

Settings

save(path)[source]

Save this Settings to a file at the given path.

Return type:

None

Logging

HTMap exposes a standard Python logging hierarchy under the logger named 'htmap'. Logging configuration can be done by any of the methods described in the documentation.

Here’s an example of how to set up basic console logging:

import logging
import sys

logger = logging.getLogger("htmap")
logger.setLevel(logging.DEBUG)

handler = logging.StreamHandler(stream=sys.stdout)
handler.setLevel(logging.DEBUG)
handler.setFormatter(
    logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
)

logger.addHandler(handler)

After executing this code, you should be able to see HTMap log messages as you tell it to do things.

Warning

The HTMap logger is not available in the context of the executing map function. Trying to use it will probably raise exceptions.

Exceptions

exception htmap.exceptions.HTMapException[source]

Base exception for all htmap exceptions.

exception htmap.exceptions.TimeoutError[source]

An operation has timed out because it took too long.

exception htmap.exceptions.MissingSetting[source]

The requested setting has not been set.

exception htmap.exceptions.OutputNotFound[source]

The requested output file does not exist.

exception htmap.exceptions.NoMapYet[source]

The htmap.MapBuilder does not have an associated htmap.Map yet because it is still inside the with block.

exception htmap.exceptions.TagAlreadyExists[source]

The requested tag already exists (recover the Map, then either use or delete it).

exception htmap.exceptions.InvalidTag[source]

The tag has an invalid character in it.

exception htmap.exceptions.TagNotFound[source]

The requested tag does not exist.

exception htmap.exceptions.EmptyMap[source]

The map contains no inputs, so it wasn’t created.

exception htmap.exceptions.ReservedOptionKeyword[source]

The map option keyword you tried to use is reserved by HTMap for internal use.

exception htmap.exceptions.MisalignedInputData[source]

There is some kind of mismatch between the lengths of the function arguments and the variadic map options.

exception htmap.exceptions.CannotRetagMap[source]

The map cannot be renamed right now.

exception htmap.exceptions.UnknownPythonDeliveryMethod[source]

The specified Python delivery method has not been registered.

exception htmap.exceptions.MapWasRemoved[source]

This map has been removed, and can no longer be interacted with.

exception htmap.exceptions.InvalidOutputStatus[source]

The output status of the map component was not recognized.

exception htmap.exceptions.MapComponentError[source]

A map component experienced an error during remote execution.

exception htmap.exceptions.MapComponentHeld[source]

A map component has been held by HTCondor.

exception htmap.exceptions.ExpectedError[source]

A map component that contained an OK result was unpacked as if it contained an error.

exception htmap.exceptions.CannotTransplantPython[source]

The Python interpreter you are using cannot be transplanted.

exception htmap.exceptions.CannotRerunComponents[source]

The given components cannot be rerun because they are currently active.

exception htmap.exceptions.InsufficientHTCondorVersion[source]

The version of HTCondor is too low to use a feature.

exception htmap.exceptions.CorruptEventLog[source]

HTMap doesn’t understand what it’s seeing in an event log.

Version

htmap.version()[source]

Return a string containing human-readable version information.

Return type:

str

htmap.version_info()[source]

Return a tuple of version information: (major, minor, micro, prerelease).

Return type:

Tuple[int, int, int, Optional[str], Optional[int]]