Map Options

Requesting Resources

The most common kind of map option you’ll probably need to work with are the ones for requests resources. HTMap makes fairly conservative default choices about the resources required by your map components. If your function needs a lot of resources, such as memory or disk space, you will need to communicate this to HTMap.

Suppose we need to transfer a huge input file that we need to read into memory, so we need a lot of memory and disk space available on the execute node. We’ll request 200 MB of RAM, 10 GB of disk space, and send our input file.

from pathlib import Path
import htmap

def read_huge_file(file):
    contents = Path(file).read_text()

    # do stuff

    return contents  # we'll just return the contents here, but imagine this is the result of processing

huge_file = htmap.TransferPath.cwd() / 'huge_file.txt'
huge_file.write_text('only a few words, but use your imagination')

42

(Don’t panic! write_text() returns the number of bytes written.)

And here’s our map call:

processed = htmap.map(
    read_huge_file,
    [huge_file],
    map_options = htmap.MapOptions(
        request_memory = '100MB',
        request_disk = '1GB',
    ),
)

print(processed.get(0))

Created map breezy-thick-beak with 1 components
only a few words, but use your imagination

request_memory and request_disk were passed as single strings. Since they are single strings, they will be treated as fixed options and applied to every component. The other kind of option is variadic, which lets you specify some option for each component of the map individually. For exampe, if we wanted a different amount of RAM for each component, we could pass a list of strings to request_memory, one for each component:

multiple = htmap.map(
    read_huge_file,
    [huge_file, huge_file, huge_file],
    map_options = htmap.MapOptions(
        request_memory = ['10MB', '20MB', '30MB'],
        request_disk = '1GB',
    ),
)
print(list(multiple))

Created map tall-soft-stream with 3 components
['only a few words, but use your imagination', 'only a few words, but use your imagination', 'only a few words, but use your imagination']

The Kitchen Sink

HTMap also supports arbitrary HTCondor submit descriptors, like you would see in a submit file. Just pass them as keyword arguments to a htmap.MapOptions, keeping in mind that you can use standard ClassAd interpolation and that the same fixed/variadic behavior applies.

If that didn’t make sense, don’t worry about it! The whole point of HTMap is to avoid needing to know too much about submit descriptors.

---

The next tutorial discusses more convenient and flexible way of defining your maps.