`pyffi.spells` — High level file operations¶

Note

This module is based on wz’s NifTester module, although nothing of wz’s original code is left in this module.

A toaster, implemented by subclasses of the abstract Toaster class, walks over all files in a folder, and applies one or more transformations on each file. Such transformations are called spells, and are implemented by subclasses of the abstract Spell class.

A spell can also run independently of a toaster and be applied on a branch directly. The recommended way of doing this is via the Spell.recurse() method.

Supported spells¶

For format specific spells, refer to the corresponding module.

Some spells are applicable on every file format, and those are documented here.

class pyffi.spells.SpellApplyPatch(toaster=None, data=None, stream=None)¶

Bases: pyffi.spells.Spell

A spell for applying a patch on files.

datainspect()¶

There is no need to read the whole file, so we apply the patch already at inspection stage, and stop the spell process by returning False.

Returns:	`False`
Return type:	`bool`

Adding new spells¶

To create new spells, derive your custom spells from the Spell class, and include them in the Toaster.SPELLS attribute of your toaster.

class pyffi.spells.Spell(toaster=None, data=None, stream=None)¶

Bases: object

Spell base class. A spell takes a data file and then does something useful with it. The main entry point for spells is recurse(), so if you are writing new spells, start with reading the documentation with recurse().

READONLY¶: A bool which determines whether the spell is read only or not. Default value is True. Override this class attribute, and set to False, when subclassing a spell that must write files back to the disk.

SPELLNAME¶: A str describing how to refer to the spell from the command line. Override this class attribute when subclassing.

data¶: The Data instance this spell acts on.

stream¶: The current file being processed.

toaster¶: The Toaster instance this spell is called from.

__init__(toaster=None, data=None, stream=None)¶

Initialize the spell data.

Parameters:	data (`Data`) – The file `data`. stream (`file`) – The file `stream`. toaster (`Toaster`) – The `toaster` this spell is called from (optional).

recurse(branch=None)¶

Helper function which calls _branchinspect() and branchinspect() on the branch, if both successful then branchentry() on the branch, and if this is succesful it calls recurse() on the branch’s children, and once all children are done, it calls branchexit().

Note that _branchinspect() and branchinspect() are not called upon first entry of this function, that is, when called with data as branch argument. Use datainspect() to stop recursion into this branch.

Do not override this function.

Parameters:	branch (`GlobalNode`) – The branch to start the recursion from, or `None` to recurse the whole tree.

_datainspect()¶

This is called after pyffi.object_models.FileFormat.Data.inspect() has been called, and before pyffi.object_models.FileFormat.Data.read() is called.

Returns:	`True` if the file must be processed, `False` otherwise.
Return type:	`bool`

datainspect()¶

This is called after pyffi.object_models.FileFormat.Data.inspect() has been called, and before pyffi.object_models.FileFormat.Data.read() is called. Override this function for customization.

Returns:	`True` if the file must be processed, `False` otherwise.
Return type:	`bool`

_branchinspect(branch)¶

Check if spell should be cast on this branch or not, based on exclude and include options passed on the command line. You should not need to override this function: if you need additional checks on whether a branch must be parsed or not, override the branchinspect() method.

Parameters:	branch (`GlobalNode`) – The branch to check.
Returns:	`True` if the branch must be processed, `False` otherwise.
Return type:	`bool`

branchinspect(branch)¶

Like _branchinspect(), but for customization: can be overridden to perform an extra inspection (the default implementation always returns True).

Parameters:	branch (`GlobalNode`) – The branch to check.
Returns:	`True` if the branch must be processed, `False` otherwise.
Return type:	`bool`

dataentry()¶

Called before all blocks are recursed. The default implementation simply returns True. You can access the data via data, and unlike in the datainspect() method, the full file has been processed at this stage.

Typically, you will override this function to perform a global operation on the file data.

Returns:	`True` if the children must be processed, `False` otherwise.
Return type:	`bool`

dataexit()¶

Called after all blocks have been processed, if dataentry() returned True.

Typically, you will override this function to perform a final spell operation, such as writing back the file in a special way, or making a summary log.

branchentry(branch)¶

Cast the spell on the given branch. First called with branch equal to data‘s children, then the grandchildren, and so on. The default implementation simply returns True.

Typically, you will override this function to perform an operation on a particular block type and/or to stop recursion at particular block types.

Parameters:	branch (`GlobalNode`) – The branch to cast the spell on.
Returns:	`True` if the children must be processed, `False` otherwise.
Return type:	`bool`

branchexit(branch)¶

Cast a spell on the given branch, after all its children, grandchildren, have been processed, if branchentry() returned True on the given branch.

Typically, you will override this function to perform a particular operation on a block type, but you rely on the fact that the children must have been processed first.

Parameters:	branch (`GlobalNode`) – The branch to cast the spell on.

classmethod toastentry(toaster)¶

Called just before the toaster starts processing all files. If it returns False, then the spell is not used. The default implementation simply returns True.

For example, if the spell only acts on a particular block type, but that block type is excluded, then you can use this function to flag that this spell can be skipped. You can also use this function to initialize statistics data to be aggregated from files, to initialize a log file, and so.

Parameters:	toaster (`Toaster`) – The toaster this spell is called from.
Returns:	`True` if the spell applies, `False` otherwise.
Return type:	`bool`

classmethod toastexit(toaster)¶

Called when the toaster has finished processing all files.

Parameters:	toaster (`Toaster`) – The toaster this spell is called from.

Grouping spells together¶

It is also possible to create composite spells, that is, spells that simply execute other spells. The following functions and classes can be used for this purpose.

pyffi.spells.SpellGroupParallel(*args)¶: Class factory for grouping spells in parallel.

pyffi.spells.SpellGroupSeries(*args)¶: Class factory for grouping spells in series.

class pyffi.spells.SpellGroupBase(toaster=None, data=None, stream=None)¶

Bases: pyffi.spells.Spell

Base class for grouping spells. This implements all the spell grouping functions that fall outside of the actual recursing (__init__(), toastentry(), _datainspect(), datainspect(), and toastexit()).

ACTIVESPELLCLASSES¶: List of active spells of this group (not instantiated). This list is automatically built when toastentry() is called.

SPELLCLASSES¶: List of Spells of this group (not instantiated).

datainspect()¶: Inspect every spell with L{Spell.datainspect} and keep those spells that must be cast.

spells¶: List of active spell instances.

classmethod toastentry(toaster)¶

classmethod toastexit(toaster)¶

class pyffi.spells.SpellGroupParallelBase(toaster=None, data=None, stream=None)¶

Bases: pyffi.spells.SpellGroupBase

Base class for running spells in parallel (that is, with only a single recursion in the tree).

branchentry(branch)¶: Run all spells.

branchexit(branch)¶

branchinspect(branch)¶: Inspect spells with Spell.branchinspect() (not all checks are executed, only keeps going until a spell inspection returns True).

changed¶

dataentry()¶: Look into every spell with Spell.dataentry().

dataexit()¶: Look into every spell with Spell.dataexit().

class pyffi.spells.SpellGroupSeriesBase(toaster=None, data=None, stream=None)¶

Bases: pyffi.spells.SpellGroupBase

Base class for running spells in series.

branchentry(branch)¶

branchinspect(branch)¶

changed¶

dataentry()¶

dataexit()¶

recurse(branch=None)¶: Recurse spells in series.

Creating toaster scripts¶

To create a new toaster script, derive your toaster from the Toaster class, and set the Toaster.FILEFORMAT attribute of your toaster to the file format class of the files it can toast.

class pyffi.spells.Toaster(spellclass=None, options=None, spellnames=None, logger=None)¶

Bases: object

Toaster base class. Toasters run spells on large quantities of files. They load each file and pass the data structure to any number of spells.

ALIASDICT¶: Dictionary with aliases for spells.

DEFAULT_OPTIONS¶: List of spell classes of the particular Toaster instance.

EXAMPLES¶: Some examples which describe typical use of the toaster.

FILEFORMAT¶: The file format class (a subclass of FileFormat).

SPELLS¶: List of all available Spell classes.

cli()¶: Command line interface: initializes spell classes and options from the command line, and run the toast() method.

exclude_types¶: Tuple of types corresponding to the exclude key of options.

get_toast_head_root_ext(filename)¶

Get the name of where the input file filename would be written to by the toaster: head, root, and extension.

Parameters:	filename (`str`) – The name of the hypothetical file to be toasted.
Returns:	The head, root, and extension of the destination, or `(None, None, None)` if `--dryrun` is specified.
Return type:	`tuple` of three `str`s

get_toast_stream(filename, test_exists=False)¶

Calls get_toast_head_root_ext(filename)() to determine the name of the toast file, and return a stream for writing accordingly.

Then return a stream where result can be written to, or in case test_exists is True, test if result would overwrite a file. More specifically, if test_exists is True, then no streams are created, and True is returned if the file already exists, and False is returned otherwise.

include_types¶: Tuple of types corresponding to the include key of options.

indent¶: An int which describes the current indentation level for messages.

inspect_filename(filename)¶: Returns whether to toast a filename or not, based on skip_regexs and only_regexs.

is_admissible_branch_class(branchtype)¶

Helper function which checks whether a given branch type should have spells cast on it or not, based in exclude and include options.

>>> from pyffi.formats.nif import NifFormat
>>> class MyToaster(Toaster):
...     FILEFORMAT = NifFormat
>>> toaster = MyToaster() # no include or exclude: all admissible
>>> toaster.is_admissible_branch_class(NifFormat.NiProperty)
True
>>> toaster.is_admissible_branch_class(NifFormat.NiNode)
True
>>> toaster.is_admissible_branch_class(NifFormat.NiAVObject)
True
>>> toaster.is_admissible_branch_class(NifFormat.NiLODNode)
True
>>> toaster.is_admissible_branch_class(NifFormat.NiMaterialProperty)
True
>>> toaster = MyToaster(options={"exclude": ["NiProperty", "NiNode"]})
>>> toaster.is_admissible_branch_class(NifFormat.NiProperty)
False
>>> toaster.is_admissible_branch_class(NifFormat.NiNode)
False
>>> toaster.is_admissible_branch_class(NifFormat.NiAVObject)
True
>>> toaster.is_admissible_branch_class(NifFormat.NiLODNode)
False
>>> toaster.is_admissible_branch_class(NifFormat.NiMaterialProperty)
False
>>> toaster = MyToaster(options={"include": ["NiProperty", "NiNode"]})
>>> toaster.is_admissible_branch_class(NifFormat.NiProperty)
True
>>> toaster.is_admissible_branch_class(NifFormat.NiNode)
True
>>> toaster.is_admissible_branch_class(NifFormat.NiAVObject)
False
>>> toaster.is_admissible_branch_class(NifFormat.NiLODNode) # NiNodes are!
True
>>> toaster.is_admissible_branch_class(NifFormat.NiMaterialProperty) # NiProperties are!
True
>>> toaster = MyToaster(options={"include": ["NiProperty", "NiNode"], "exclude": ["NiMaterialProperty", "NiLODNode"]})
>>> toaster.is_admissible_branch_class(NifFormat.NiProperty)
True
>>> toaster.is_admissible_branch_class(NifFormat.NiNode)
True
>>> toaster.is_admissible_branch_class(NifFormat.NiAVObject)
False
>>> toaster.is_admissible_branch_class(NifFormat.NiLODNode)
False
>>> toaster.is_admissible_branch_class(NifFormat.NiSwitchNode)
True
>>> toaster.is_admissible_branch_class(NifFormat.NiMaterialProperty)
False
>>> toaster.is_admissible_branch_class(NifFormat.NiAlphaProperty)
True

logger¶: A logging.Logger for toaster log messages.

msg(message)¶

Write log message with logger.info(), taking into account indent.

Parameters:	message (`str`) – The message to write.

msgblockbegin(message)¶: Acts like msg(), but also increases indent after writing the message.

msgblockend(message=None)¶: Acts like msg(), but also decreases indent before writing the message, but if the message argument is None, then no message is printed.

only_regexs¶: Tuple of regular expressions corresponding to the only key of options.

options¶: The options of the toaster, as dict.

static parse_inifile(option, opt, value, parser, toaster=None)¶

Initializes spell classes and options from an ini file.

>>> import pyffi.spells.nif
>>> import pyffi.spells.nif.modify
>>> class NifToaster(pyffi.spells.nif.NifToaster):
...     SPELLS = [pyffi.spells.nif.modify.SpellDelBranches]
>>> import tempfile
>>> cfg = tempfile.NamedTemporaryFile(delete=False)
>>> cfg.write("[main]\n")
>>> cfg.write("spell = modify_delbranches\n")
>>> cfg.write("folder = tests/nif/test_vertexcolor.nif\n")
>>> cfg.write("[options]\n")
>>> cfg.write("source-dir = tests/\n")
>>> cfg.write("dest-dir = _tests/\n")
>>> cfg.write("exclude = NiVertexColorProperty NiStencilProperty\n")
>>> cfg.write("skip = 'testing quoted string'    normal_string\n")
>>> cfg.close()
>>> toaster = NifToaster(logger=fake_logger)
>>> import sys
>>> sys.argv = [
...     "niftoaster.py",
...     "--ini-file=utilities/toaster/default.ini",
...     "--ini-file=%s" % cfg.name,
...     "--noninteractive", "--jobs=1"]
>>> toaster.cli()
pyffi.toaster:INFO:=== tests/nif/test_vertexcolor.nif ===
pyffi.toaster:INFO:  --- modify_delbranches ---
pyffi.toaster:INFO:    ~~~ NiNode [Scene Root] ~~~
pyffi.toaster:INFO:      ~~~ NiTriStrips [Cube] ~~~
pyffi.toaster:INFO:        ~~~ NiStencilProperty [] ~~~
pyffi.toaster:INFO:          stripping this branch
pyffi.toaster:INFO:        ~~~ NiSpecularProperty [] ~~~
pyffi.toaster:INFO:        ~~~ NiMaterialProperty [Material] ~~~
pyffi.toaster:INFO:        ~~~ NiVertexColorProperty [] ~~~
pyffi.toaster:INFO:          stripping this branch
pyffi.toaster:INFO:        ~~~ NiTriStripsData [] ~~~
pyffi.toaster:INFO:creating destination path _tests/nif
pyffi.toaster:INFO:  writing _tests/nif/test_vertexcolor.nif
pyffi.toaster:INFO:Finished.
>>> import os
>>> os.remove(cfg.name)
>>> os.remove("_tests/nif/test_vertexcolor.nif")
>>> os.rmdir("_tests/nif/")
>>> os.rmdir("_tests/")
>>> for name, value in sorted(toaster.options.items()):
...     print("%s: %s" % (name, value))
applypatch: False
archives: False
arg: 
createpatch: False
destdir: _tests/
diffcmd: 
dryrun: False
examples: False
exclude: ['NiVertexColorProperty', 'NiStencilProperty']
helpspell: False
include: []
inifile: 
interactive: False
jobs: 1
only: []
patchcmd: 
pause: True
prefix: 
raisetesterror: False
refresh: 32
resume: True
series: False
skip: ['testing quoted string', 'normal_string']
sourcedir: tests/
spells: False
suffix: 
verbose: 1

skip_regexs¶: Tuple of regular expressions corresponding to the skip key of options.

spellnames¶: A list of the names of the spells.

toast(top)¶

Walk over all files in a directory tree and cast spells on every file.

Parameters:	top (str) – The directory or file to toast.

toast_archives(top)¶: Toast all files in all archives.

top¶: Name of the top folder to toast.

write(stream, data)¶: Writes the data to data and raises an exception if the write fails, but restores file if fails on overwrite.

writepatch(stream, data)¶: Creates a binary patch for the updated file.

`pyffi.spells` — High level file operations¶

Supported spells¶

Adding new spells¶

Grouping spells together¶

Creating toaster scripts¶

Table Of Contents

This Page

Navigation

pyffi.spells — High level file operations¶

Supported spells¶

Adding new spells¶

Grouping spells together¶

Creating toaster scripts¶

Table Of Contents

This Page

Navigation

`pyffi.spells` — High level file operations¶