…rors`

Updating dependencies to bring in new libraries such as locks, resources and errors. Some domains have been updated to use this change.

Updated pkgs.nix to a5774e76bb8c3145eac524be62375c937143b80c
Updated node2nix to 1.11.0 and loading directly from GitHub
Updated pkg to 5.6.0 and using pkg-fetch 3.3 base binaries
Updated to TypeScript 4.5+
Updated @types/node to 16.11.7
Updated node-gyp-build to 4.4.0
Updated typedoc to 0.22.15, the .nojekyll generation is automatic
Changed to target ES2021 as node 16 supports it
Bin executable duct-tape in default.nix because node2nix no longer generates bin executables
Updated pkg builds hashes to v16.14.2 and specified target node range

Updated js-async-init from 1.6.0 to 1.7.1
Updated js-async-locks from 2.0.0 to 2.2.0
Updated js-db from 3.2.0 to 3.2.3
Updating js-workers from 1.2.5 to 1.3.1

The `data` POJO that was originally supplied in the constructor as the second parameter is now part of the `options` parameter and most places in the code have been updated to match this.

#304

When any error is thrown as the result of another error occurring, the original error is now contained within the `cause` property of the new error.

#304

#304

…or chaining

Our gRPC `toError` and `fromError` utils are now able to serialise and deserialise Polykey and non-Polykey errors (as well as non-errors), including the entire error chain if this exists. Also includes the ability to filter out sensitive data, for example when the error is being sent to another agent.

Errors sent over the network in this way are now additionally wrapped on the receiving side in an `ErrorPolykeyRemote` to make the source of the error more clear.

#304

All of the agent and client service handlers are now passed a shared logger that is used to log errors to stderr.

#304

Note that the `transaction` and `_transaction` methods still need te removed.

I opted not to use transactions for the most part. Adding things to the queue has to be handled with locking anyway so I opted to rely on that istead. This still bears testing though.

This was throwing an error because the expected object structure changed. The typing is brittle here since we're going around typescript's back to get a private property.

Sequence numbers and claim ids needed locking due to serialisation concerns

Notifications have strict ordering in the DB, therefore locking is required.

There are still some tests failing due to cloning and pulling. Seems like a problem with the HTTP API for cloning.

There was a bug with the `fromJSON` method where it couldn't deserialise an error with an undefined cause (because this is removed by the replacer).

…ors properly

Making use of the updated API to use the error provided to the resourceRelease.

… locking

Using `LockBox` allowed us to greatly simplify using the object map and creating connections.

Using `LockBox` allowed us to greatly simplify using the object map and creating vaults.

Specifying `--format=json` will cause errors to be logged in json format. ErrorPolykeyRemote errors also now contain additional metadata about the origin node of the error.

#323

Had to fix `expectProcessError` to work with the raw JSON forms of the errors rather than attempting to parse the errors. This is due to the bin CLI errors being separate from the normal errors and the reviver being unable to revive them.

I also had to fix how the `ErrorPolykeyRemote` converted the metadata with toJSON/fromJSON.

There was a bug with the DB that is getting fixed. This just throws a clear error when a notification can't be read despite it being in the list of all notifications.

Problem was with the tests themselves so i've fixed the tests. If there are any similar issues crop up later, then we can explore the underlying cause in depth. However, it is low priority for now.

There were places where ClientMetadata was not being provided to grpc calls (`{} as ClientMetadata`) however this information is required in order to construct ErrorPolykeyRemote. Where metadata is needed in tests it is either now taken from constants used in the tests or mocked. There was also an issue with error descriptions not being serialised/deserialised which is now fixed. Also some brittle test cases for error messages were removed.

Had to fix the usage in 5 domains.

It was just a problem with trying to connect to '0.0.0.0' with the UTP socket.

Having `ClientMetadata` in the top-level types caused an import loop. By moving it and `ErrorPolykeyRemote` to the grpc domain this loop is resolved. This was the best option since now all usage of `ClientMetadata` is inside the grpc domain.

#323

We only want to log out server errors on the server side, so we now apply filters on errors before logging them.

Sometimes an error is only considered a client error when part of the chain of another error (for example ErrorPolykeyRemote) so it is now possible to specify this array.

#304

Retrieves all buckets from the NodeGraph

- added getBuckets test for distance and lastUpdated order
- resetting buckets work
- changing utility names

Implemented `getClosestNodes()` and relevant tests in `NodeGraph.test.ts`.

Relates to #212

…et is full

Logic of adding nodes has been split between `NodeManager` and `NodeGraph`. The `NodeGraph.setNode` just handles adding a node to the bucket where the `NodeManager.setNode` contains the logic of when to add the node

Relates #359

fix into 7cc74d0

Added a callback to the `Proxy` that is called when a `ForwardConnection` or `ReverseConnection` is established and authenticated. It is called with the following connection information; `remoteNodeId`, `remoteHost`, `remotePort` and `type`. They type signifies if it was a forward or reverse connection. Note that this is only triggered by composed connections.

Added a test for if the callback was called when a `ReverseConnection` is established.

Relates #332
Relates #344

Added an event to the `EventBus` that is triggered by the `Proxy`'s `connectionEstablishedCallback`. this adds the node to the `NodeGraph`.

Related #344

In some cases we want to specify how long we attempt to connect to a node on a per-connection basis.

Related #363

Added `nodePing` command to `NodeConnectionManager` and `NodeManager.nodePing` calls that now. the new `nodePing` command essentially attempts to create a connection to the target node. In this process we authenticate the connection and check that the nodeIds match. If no address is provided it will default to trying to find the node through kademlia.

Related #322

`setNode` now authenticates the node you are trying to add. Added a flag for skipping this authentication as well as a timeout timer for the authentication. this is shared between authentication new node and the old node if the bucket is full.

Related #322

Updated `NodeConnectionManager.pingNode` to just use the proxy connection.

#322

#322

`setNode` now pings 3 nodes concurrently, updating ones that respond and removing ones that don't. If there is room in the bucket afterwards then we add the new node.

#322

#322

`setNode` now has a `blocking` flag that defaults to false. If it encounters a full bucket when adding a node then it will add the operation to the queue and asynchronously trys a blocking `setNode` in the background. `setNode`s will only be added to the queue if the bucket was full.

#322

`NodeConnectionManager` now takes `NodeGraph` in  the `nodeConnectionManager.start` method. It has to be part of the start method since they are co-dependent. `NodeConnectionManager` cals `NodeManager.setNode()` when a connection is established. This fulfills the condition of adding a node to the graph during a forward connection.

Fixed up tests that were failing in relation to the `NodeManager` `StartStop` conversion.

#322

This method preforms the kademlia `refreshBucket` operation. It selects a random node within the bucket and preforms a search for that node. The process exchanges node information with any nodes it connects to.

#345

Added queuing for `refreshBucket`. This means that buckets will be refreshed one at a time sequentially. This is to avoid doing a lot of costly refreshing all at once.

Added no activity for buckets. If a bucket hasn't been touched for a while, 1 hour by default, it will add a refresh bucket operation to the queue. Timers are disabled for buckets already in the queue. Only 1 timer is used for all buckets since only one of them can have the shortest timer and that's all we really care about.

#345

`nodeConnectionManager.syncNodeGraph` now refreshes all buckets above the closest node as per the kademlia spec. This means adding a lot of buckets to the refresh bucket queue when an agent is started.

#345

Added support to cancel out of a `refreshBucket` operation. This is to allow faster stopping of the `NodeManager` by aborting out of a slow `refreshBucket` operation. This has been implemented with the `AbortController`/`AbortSignal` API. This is not fully supported by Node14 so we're using the `node-abort-controller` to provide functionality for now.

#345

#322

`NodeManager.setNode` and `NodeConnectionManager.syncNodeGraph` now utilise a single, shared queue to asynchronously add nodes to the node graph without blocking the main loop. These methods are both blocking by default but can be made non-blocking by setting the `block` parameter to false.
#322

#322

Renamed `queueStart` and `queuePush` since `Queue` is its own class now
Simplified some logic using the `promise` utility

This contains fixes for failing tests as well as fixes for tests failing to exit when finished.

This checks if we await things that are not promises. This is not a problem per se, but we generally don't want to await random things.