Module parse2 

New netdoc parsing arrangements, with derive

Parsing principles

A parseable network document is a type implementing NetdocParseable. usually via the NetdocParseable derive=deftly macro`.

A document type is responsible for recognising its own heading item. Its parser will also be told other of structural items that it should not consume. The structural lines can then be used to pass control to the appropriate parser.

A “structural item” is a netdoc item that is defines the structure of the document. This includes the intro items for whole documents, the items that introduce document sections (which we model by treating the section as a sub-document) and signature items (which introduce the signatures at the end of the document, and after which no non-signature items may appear).

Ordering

We don’t always parse things into a sorted order. Sorting will be done when assembling documents, before outputting.

Types, and signature handling

Most top-level network documents are signed somehow. In this case there are three types:

• FooUnverified: a signed Foo, with its signatures, not yet verified. Implements NetdocUnverified, typically by invoking the NetdocUParseablenverified derive macro on Foo.

  Type-specific methods are provided for verification, to obtain a Foo.

• Foo: the body data for the document. This doesn’t contain any signatures. Having one of these to play with means signatures have already been validated. Can be parsed as part of the signed document, via the NetdocParseable implementation on FooUnverified, and then obtained via .verify_... method(s) on FooUnverified,

• FooSignatures: the signatures for a Foo. Implements NetdocParseableSignatures, via derive, with #[deftly(netdoc(signatures))].

Relationship to tor_netdoc::parse

This is a completely new parsing approach, based on different principles. The key principle is the recognition of “structural keywords”, recursively within a parsing stack, via the pNetdocParseable] trait.

This allows the parser to be derived. We have type-driven parsing of whole Documents, Items, and their Arguments and Objects, including of their multiplicity.

The different keyword handling means we can’t use most of the existing lexer, and need new item parsing API:

• NetdocParseable trait.
• KeywordRef type.
• ItemStream, UnparsedItem, ArgumentStream, UnparsedObject.

The different error handling means we have our own error types. (The crate’s existing parse errors have information that we don’t track, and is also a portmanteau error for parsing, writing, and other functions.)

Document signing is handled in a more abstract way.

Some old netdoc constructs are not supported. For example, the obsolete opt prefix on safe-to-ignore Items. The parser may make different decisions about netdocs with anomalous item ordering.

Re-exports

pub use keyword::KeywordRef;

Modules

derive 🔒

Deriving NetdocParseable

error 🔒

Parsing errors

impls 🔒

Implementations of our useful traits, on external and parsing-mode types

keyword

Keywords in netdocs

lex 🔒

Lexing of netdoc elements

lines 🔒

Version of std::str::Lines that tracks line numbers and has remainder()

multiplicity

Multiplicity of fields (Items and Arguments)

poc

Proof-of-concept parser using parse2 for network status documents.

sig_hashes

Hash types suitable for use as #[deftly(netdoc(signature(hash_accu = TY))]

signatures 🔒

Handling of netdoc signatures

structural 🔒

Structural keyword recognition helpers

traits 🔒

Core model for netdoc parsing

Structs

ArgumentStream

Reader for arguments on an Item

IsStructural

Token indicating that a keyword is structural

ItemStream

Top-level reader: Netdoc text interpreted as a stream of items

Lines

Version of std::str::Lines that tracks line numbers and has remainder()

NoFurtherArguments

End of an argument list that does not accept any further (unknown) arguments

ParseError

Error encountered when parsing a document, including its location

ParseInput

Input to a network document top-level parsing operation

ParseOptions

Options for parsing

Peeked

Information about the next line we have peeked

SignatureHashInputs

Inputs needed to calculate a specific signature hash for a specific Item

SignaturesData

The signatures information extracted from a signed network document

StopAt

Predicate for testing whether a keyword is a structural one that we should stop at

UnexpectedArgument

An unexpected argument was encountered

UnparsedItem

An Item that has been lexed but not parsed

UnparsedObject

An Object that has been lexed but not parsed

Enums

ArgumentError

Problem found when parsing an individual argument in a netdoc keyword item

ErrorProblem

Problem found when parsing a document

VerifyFailed

Error from signature verification (and timeliness check)

Traits

HasUnverifiedParsedBody

Network document that has an unparsed body type (internal trait)

ItemArgumentParseable

An (individual) argument that can be parsed from in a netdoc

ItemObjectParseable

An Object value that be parsed from a netdoc

ItemValueParseable

An item (value) that can be parsed in a netdoc

NetdocParseable

A document or section that can be parsed

NetdocParseableFields

A collection of fields that can be parsed within a section

NetdocParseableSignatures

The signatures section of a network document, that can be parsed

NetdocUnverified

A network document with (unverified) signatures

SignatureHashesAccumulator

Hash(es) for a signature item

SignatureItemParseable

A signature item that can appear in a netdoc

StopPredicate

Raw predicate, usually a closure, that can appear within StopAt.

StrExt

Extension trait adding a method to str

Functions

check_validity_time

Utility function to check that a time is within a validity period

check_validity_time_tolerance

Like check_validity_time() but with a tolerance to support clock skews.

parse_internal 🔒

Common code for parse_netdoc and parse_netdoc_multiple

parse_netdoc

Parse a network document - toplevel entrypoint

parse_netdoc_multiple

Parse multiple concatenated network documents - toplevel entrypoint

parse_netdoc_multiple_with_offsets

Parse multiple network documents, also returning their offsets - toplevel entrypoint

Type Aliases

NdaSystemTimeDeprecatedSyntax

Date and time in deprecated ISO8601-with-space separate arguments syntax