Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introducing vault sharing and the restrictions of pulling and cloning with vault permissions #262

Closed
wants to merge 124 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
124 commits
Select commit Hold shift + click to select a range
fde6610
WIP
CMCDragonkai Sep 14, 2021
7a1c946
WIP - Updating EFS version
tegefaulkes Sep 17, 2021
61d0065
WIP - Fixes to for DB changes.
tegefaulkes Sep 20, 2021
a441159
WIP - coverting PolykeyAgent to use the `createPolykeyAgent` style of…
tegefaulkes Sep 20, 2021
3cd67aa
WIP - reverted some changes and fixed some code.
tegefaulkes Sep 21, 2021
0d0fa41
wip: expanding testing for git utility functions
Sep 21, 2021
66f3711
Fixed tests.
tegefaulkes Sep 21, 2021
36c8bec
WIP - switching NodeId to base58
tegefaulkes Sep 21, 2021
9d86b29
makeNodeId throws a useful error now.
tegefaulkes Sep 22, 2021
1ddf039
Added nodeID type guards to the client RPCs
tegefaulkes Sep 22, 2021
cdda7ea
Updated tests to use the NodeId type guards.
tegefaulkes Sep 22, 2021
74e65d8
Linting fixes.
tegefaulkes Sep 22, 2021
3cc7916
Integrated `@matrixai/workers`
tegefaulkes Sep 23, 2021
4de61ac
`@matrixai/db` Integraton.
tegefaulkes Sep 23, 2021
137ac9f
Small fix for lint errors.
tegefaulkes Sep 23, 2021
8b8173f
DB - Any use of a lexi-int as a DB key was converted to a buffer.
tegefaulkes Sep 24, 2021
716514f
Moved DB key handling into the `KeyManager`
tegefaulkes Sep 24, 2021
d6f56c8
WIP - getting polykey to start with encryption
tegefaulkes Sep 27, 2021
e28c097
DB starting with encryption now.
tegefaulkes Sep 27, 2021
0dcdb67
Fleshing out VaultInternal
Sep 27, 2021
f68e8da
Minor fixes for tests.
tegefaulkes Sep 27, 2021
cb865bb
Removed old DB code.
tegefaulkes Sep 27, 2021
d1f34cc
Vault Internal class
Sep 28, 2021
b62f143
versionCheckout added
Sep 29, 2021
7ecb048
Added some DBkey tests to keymanager.
tegefaulkes Sep 30, 2021
f0e5647
Main DB and EFS DBs now use the workerManger.
tegefaulkes Sep 30, 2021
0221ca7
Tests: changing all .rejects.toThrow() to arrow functions to preserve…
joshuakarp Oct 1, 2021
2394670
Converted Domains to using `create/destroy + start/stop` patterns fro…
tegefaulkes Sep 28, 2021
4a02210
Vault Manager lifecycle management
Oct 1, 2021
1431e99
re-enabled vaults tests.
tegefaulkes Oct 1, 2021
f81cb5b
Transaction for vault lifecycle
Oct 4, 2021
0fc8756
Access function for Vault Internal
Oct 5, 2021
cb33b5c
Restructuring old and new Vault domain classes
Oct 5, 2021
d8d8256
Set 1 core for tests using a WorkerManager.
tegefaulkes Oct 5, 2021
9dc2e6b
Converted VaultManager and VaultInternal to using js-async-init
tegefaulkes Oct 5, 2021
5e24e03
WIP: Vault checkout testing
Oct 5, 2021
4fa8256
Making VaultsInternal purely create/destroy.
tegefaulkes Oct 6, 2021
66af390
Added VaultInternal tests
tegefaulkes Oct 6, 2021
bad79bc
WIP: Vault Manager tests
Oct 6, 2021
86f19b6
Updated VaultInternal.test.ts read and write tests.
tegefaulkes Oct 7, 2021
0667325
Added vaultsVersion command to CLI.
tegefaulkes Oct 7, 2021
bc705f1
NodeConnectionMap type added (identical to VaultMap)
joshuakarp Oct 8, 2021
a86be96
Implementing vaults version tests, fixing tests, other fixes.
tegefaulkes Oct 8, 2021
a04a1ec
Fleshing out more vaults version tests.
tegefaulkes Oct 8, 2021
4218d1f
Fixing sessions bin tests (using new init style)
joshuakarp Oct 11, 2021
11d6d30
Adding test fixes from vaultsrefactoring_fixing_tests (mostly for Nod…
joshuakarp Oct 11, 2021
c9f0d68
Finished off vaults version CLI.
tegefaulkes Oct 11, 2021
58fa4f6
Added a `vaultKey` to the `keyManager`. updated every instance of `Va…
tegefaulkes Oct 11, 2021
4250bb3
Small fix for VaultManager destroying properly.
tegefaulkes Oct 11, 2021
1d14e20
Using status matrix
Oct 11, 2021
fc77f91
Small fixes to bootstrap and polykey.
tegefaulkes Oct 11, 2021
36d72ef
Updating to encryptedfs 3.2.0
Oct 11, 2021
8506dbb
general test fixes.
tegefaulkes Oct 11, 2021
881aa6b
Small fixes to `clientService.test.ts` vault tests.
tegefaulkes Oct 11, 2021
af1b4e3
Added a `VaultFacade` type, public methods on VaultManager now return…
tegefaulkes Oct 12, 2021
c5de764
Expanding Vault Manager tests
Oct 12, 2021
b0d8374
Fixing Vault Facade types
Oct 12, 2021
92b90e5
NodeConnection: implementing locking on instantiation, changing to ge…
joshuakarp Oct 12, 2021
a8ca8ee
Undoing test.only
joshuakarp Oct 12, 2021
6c58ba4
changing vault map
Oct 12, 2021
fabfe82
WIP - Integrating `@matrixai/db`
tegefaulkes Oct 12, 2021
c9d9ca7
small changes;
tegefaulkes Oct 13, 2021
686cac4
Converted Permission ID to using `@matrixai/id`
tegefaulkes Oct 13, 2021
6ea998e
WIP - NotificationID conversion to using js-id.
tegefaulkes Oct 13, 2021
b0c84cc
git utils testing fixes
Oct 13, 2021
a7c2665
NotificationID converted to using js-id.
tegefaulkes Oct 13, 2021
681f344
General fixes
tegefaulkes Oct 13, 2021
8fc4003
Converted ClaimId to using IdSortable.
tegefaulkes Oct 14, 2021
fc7993b
NodeManager: getNodeConnection tests (basic, existing, concurrent)
joshuakarp Oct 15, 2021
d5fac41
Small efficiency increase: don't need to resolve node ID -> node addr…
joshuakarp Oct 15, 2021
b405ac8
Implemented VaultOps + tests.
tegefaulkes Oct 14, 2021
f08c2c2
Fixes to clientService.test.ts vaults tests
tegefaulkes Oct 15, 2021
3f4c549
Fixes to secrets CLI tests.
tegefaulkes Oct 15, 2021
7a01bf4
Checking for 'end' tag done inside the VaultInternal now instead of r…
tegefaulkes Oct 15, 2021
b106d3a
Fixing shutdown issue with NodeManager cross-signing claims test
joshuakarp Oct 18, 2021
b907028
Adding return type to claimNode in NodeConnection
joshuakarp Oct 18, 2021
7dceaf0
Using js-id `v2.0.0`, updated Ids with changes.
tegefaulkes Oct 18, 2021
a732e5e
Added tests to check if `vault.commit()` and `vault.access()` properl…
tegefaulkes Oct 18, 2021
3c50978
Vault sharing and cloning
Oct 18, 2021
debc0db
wip
Oct 18, 2021
fc84b7d
WIP
Oct 18, 2021
9d845ee
PolykeyAgent: Injecting GestaltGraph and ACL in VaultManager
joshuakarp Oct 19, 2021
d99e070
Fixing up Vault Manager testing with ids
Oct 19, 2021
3b62c08
Restructuring Git Request class
Oct 19, 2021
ee38c3b
Log and log format.
tegefaulkes Oct 19, 2021
c589682
Pull vaults and cloning fixes
Oct 19, 2021
292be71
Removed `lexicographic-integer` and `monotonic-lexicographic-timestam…
tegefaulkes Oct 19, 2021
f3788ce
When commiting to a vault the author.name is now the NodeId instead o…
tegefaulkes Oct 20, 2021
71a33eb
Added `lexicographic-integer`, Removed `numToLexiString` and `lexiStr…
tegefaulkes Oct 20, 2021
a3d5a31
Small fix to vaults log CLI command.
tegefaulkes Oct 20, 2021
e535f5e
Added checks for the lock being held in the commit and access tests.
tegefaulkes Oct 20, 2021
f863414
Replaced vaultInternal 'end' tag with 'last'.
tegefaulkes Oct 20, 2021
f010ae7
Added checks to file contents in `Commit message contains all actions…
tegefaulkes Oct 20, 2021
cc27c29
Disabling failing session test: check parallel token refreshing
joshuakarp Oct 21, 2021
e711c04
Small fixes.
tegefaulkes Oct 21, 2021
9b853dd
Changing version message to a comment
joshuakarp Oct 21, 2021
c363733
1st pass linting.
tegefaulkes Oct 21, 2021
bd243a5
Notifications test fixes - all passing
joshuakarp Oct 21, 2021
057cc60
Increasing beforeAll test timeout for NodeManger keynode creation
joshuakarp Oct 21, 2021
1e6d49f
WIP - test fixes
tegefaulkes Oct 21, 2021
8655b87
Updated to js-id 2.1.0 due to IdSortable bug
CMCDragonkai Oct 21, 2021
fcbf3b5
GRPCClientAgent.test.ts: fixing can successfully sign a claim
joshuakarp Oct 21, 2021
3cbb0f2
GRPCClientAgent.test.ts: fixing logger output
joshuakarp Oct 21, 2021
b9a1cca
PolykeyAgent can now be started without a `WorkerManager` by setting …
tegefaulkes Oct 22, 2021
8eff7e0
Fixing vault version client tests
Oct 22, 2021
a5b19e9
NodeIds are now base32hex + other small fixes.
tegefaulkes Oct 22, 2021
fc80ea1
NodeGraph: changing number of node ID bits (for base32hex ID)
joshuakarp Oct 25, 2021
fa1b5dc
isVaultIdPretty typo
joshuakarp Oct 25, 2021
84edbd9
WIP - NodeId fixes.
tegefaulkes Oct 25, 2021
71fee20
Nodes utils tests fixed: all passing using 32 byte node ID
joshuakarp Oct 25, 2021
3890e36
Utils fix
joshuakarp Oct 25, 2021
72dee5a
NodeGraph test fixes: using 32 byte node ID
joshuakarp Oct 25, 2021
addaaeb
console.log remove
joshuakarp Oct 25, 2021
715b42b
WIP - NodeId fixes.
tegefaulkes Oct 25, 2021
9961e4f
Fixed a mistake.
tegefaulkes Oct 25, 2021
301e2b6
NodeConnection tests fix: correct shutdown now by changing order of d…
joshuakarp Oct 25, 2021
86e23ea
Changing order of shutdown in other tests
joshuakarp Oct 25, 2021
9d77557
PolykeyAgent shutdown order cleanup
joshuakarp Oct 25, 2021
ea19aa7
fix for reset index on unmodified files in git
Oct 25, 2021
e879b93
Linting
tegefaulkes Oct 25, 2021
d9fe886
All tests are passing.
tegefaulkes Oct 25, 2021
7a3a10c
Vaults permissions
Oct 25, 2021
ad47dc0
wip
Oct 25, 2021
9e0cd03
WIP
Oct 25, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions .eslintrc
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@
"commonjs": true,
"es6": true,
"node": true,
"jest": true,
"jasmine": true
"jest": true
},
"parser": "@typescript-eslint/parser",
"extends": [
Expand All @@ -19,6 +18,9 @@
"sourceType": "module",
"ecmaVersion": 2020
},
"globals": {
"fail": "readonly"
},
"rules": {
"linebreak-style": ["error", "unix"],
"no-empty": 1,
Expand All @@ -40,13 +42,20 @@
"@typescript-eslint/no-namespace": 0,
"@typescript-eslint/no-explicit-any": 0,
"@typescript-eslint/explicit-module-boundary-types": 0,
"@typescript-eslint/no-unused-vars": "warn",
"@typescript-eslint/no-unused-vars": [
"warn",
{
"varsIgnorePattern": "^_",
"argsIgnorePattern": "^_"
}
],
"@typescript-eslint/no-inferrable-types": 0,
"@typescript-eslint/no-non-null-assertion": 0,
"@typescript-eslint/no-this-alias": 0,
"@typescript-eslint/no-var-requires": 0,
"@typescript-eslint/ban-ts-comment": 0,
"@typescript-eslint/no-empty-function": 0,
"@typescript-eslint/no-empty-interface": 0,
"@typescript-eslint/naming-convention": [
"error",
{
Expand All @@ -64,6 +73,7 @@
{
"selector": "parameter",
"format": ["camelCase"],
"leadingUnderscore": "allow",
"trailingUnderscore": "allowSingleOrDouble"
},
{
Expand Down
2,979 changes: 1,394 additions & 1,585 deletions package-lock.json

Large diffs are not rendered by default.

13 changes: 8 additions & 5 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,12 @@
"polykey": "ts-node -r tsconfig-paths/register src/bin/polykey.ts"
},
"dependencies": {
"@grpc/grpc-js": "^1.2.5",
"@grpc/grpc-js": "1.3.7",
"@matrixai/async-init": "^1.0.0",
"@matrixai/db": "^1.0.1",
"@matrixai/id": "^2.1.0",
"@matrixai/logger": "^2.0.1",
"@matrixai/workers": "^1.2.3",
"ajv": "^7.0.4",
"async-mutex": "^0.2.4",
"bip39": "^3.0.3",
Expand All @@ -79,7 +83,7 @@
"commander": "^7.1.0",
"cross-fetch": "^3.0.6",
"cross-spawn": "^7.0.3",
"encryptedfs": "^2.1.3",
"encryptedfs": "^3.2.0",
"fast-fuzzy": "^1.10.8",
"fd-lock": "^1.2.0",
"google-protobuf": "^3.14.0",
Expand All @@ -88,7 +92,6 @@
"jose": "^3.12.3",
"level": "^7.0.0",
"lexicographic-integer": "^1.1.0",
"monotonic-lexicographic-timestamp": "^1.0.0",
"node-forge": "^0.10.0",
"pako": "^1.0.11",
"prompts": "^2.4.1",
Expand All @@ -99,8 +102,8 @@
"threads": "^1.6.5",
"ts-custom-error": "^3.2.0",
"utp-native": "^2.5.3",
"uuid": "^8.3.0",
"virtualfs": "^2.2.0"
"multiformats": "^9.4.8",
"uuid": "^8.3.0"
},
"devDependencies": {
"@babel/preset-env": "^7.13.10",
Expand Down
119 changes: 119 additions & 0 deletions src/GenericIdTypes.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
import { utils as idUtils } from '@matrixai/id';
import { ErrorInvalidId } from './errors';
import { Id as InternalId } from '@matrixai/id/dist/Id';
import type { Codec } from 'multiformats/bases/base';
import { bases } from 'multiformats/basics';

type MultibaseFormats = keyof typeof bases;
/// This is the internal form of the Id.
export type Id = InternalId;
/// This is the user readable string form of the Id.
export type IdString = string;
// This is the number of bytes a valid Id has
const idValidByteLength = 16;

// Type guards for generic RandomId types.
function isId<T extends Id>(arg: any): arg is T {
if (!(arg instanceof Uint8Array)) return false;
return arg.length === idValidByteLength;
}

/**
* This will return arg as a valid VaultId or throw an error if it can't be converted.
* This will take a multibase string of the ID or the raw Buffer of the ID.
* @param arg - The variable we wish to convert
* @throws vaultErrors.ErrorInvalidVaultId if the arg can't be converted into a VaultId
* @returns VaultIdRaw
*/
function makeId<T extends Id>(arg: any): T {
let id = arg;
// Checking and converting a string
if (typeof arg === 'string') {
// Covert the string to the Buffer form.
try {
id = idUtils.fromMultibase(arg);
if (id == null) throw new ErrorInvalidId();
} catch (err) {
throw new ErrorInvalidId();
}
}

// If its a buffer we convert it to a Id.
if (arg instanceof Buffer) id = idUtils.fromBuffer(id);

// Checking if valid buffer.
if (isId<T>(id)) return id;
throw new ErrorInvalidId();
}

function isIdString<T extends IdString>(
arg: any,
validByteLength: number = idValidByteLength,
): arg is T {
if (typeof arg !== 'string') return false;
const id = fromMultibase(arg);
if (id == null) return false;
return id.length === validByteLength;
}

function makeIdString<T extends IdString>(
arg: any,
validByteLength: number = idValidByteLength,
format: MultibaseFormats = 'base58btc',
): T {
const id = arg;
if (id instanceof Uint8Array) {
if (id.length !== validByteLength) throw new ErrorInvalidId();
return toMultibase(arg, format) as T;
}
if (isIdString<T>(id, validByteLength)) return id;
throw new ErrorInvalidId();
}

function idToString(id: Id): IdString {
return id.toString();
}

function stringToId(idString: IdString): Id {
return idUtils.fromString(idString)!;
}

// Multibase helper functions.
const basesByPrefix: Record<string, Codec<string, string>> = {};
for (const k in bases) {
const codec = bases[k];
basesByPrefix[codec.prefix] = codec;
}

/**
* Encodes an multibase ID string
*/
function toMultibase(id: Uint8Array, format: MultibaseFormats): string {
const codec = bases[format];
return codec.encode(id);
}

/**
* Decodes a multibase encoded ID
* Do not use this for generic multibase strings
*/
function fromMultibase(idString: string): Uint8Array | undefined {
const prefix = idString[0];
const codec = basesByPrefix[prefix];
if (codec == null) {
return;
}
const buffer = codec.decode(idString);
return new Uint8Array(buffer);
}

export {
isId,
makeId,
isIdString,
makeIdString,
idToString,
stringToId,
toMultibase,
fromMultibase,
};
Loading