Indexer code #4

lachlanglen · 2024-03-21T17:35:48Z

No description provided.

schema.sql

lachlanglen · 2024-03-21T17:42:50Z

schema.sql

+
+CREATE TABLE
+  list_registration (
+    id SERIAL PRIMARY KEY,


change this to INT rather than SERIAL as we will want to manually set it as per registration ID on list contract (which will be an int, but may not increment identically and so we want to make sure there are no discrepancies)

registration ID is not returned in the register action, the id in the action result is the project id provided by the user, or the signerID.

Isn't the Registration returned in the new Lists contract? I think since the Lists contract will be out within a day or so and all existing projects on registry.potlock.near will be migrated there via the register method, we don't have to account for the current registry contract.

schema.sql

lachlanglen · 2024-03-21T17:54:00Z

schema.sql

+
+CREATE TABLE
+  pot (
+    id VARCHAR PRIMARY KEY,


change to SERIAL

lachlanglen · 2024-03-21T17:54:34Z

schema.sql

+    FOREIGN KEY (owner_id) REFERENCES account (id),
+    FOREIGN KEY (chef_id) REFERENCES account (id),
+    FOREIGN KEY (base_currency) REFERENCES account (id)
+  );


add:

cooldown_period_ms INT NOT NULL
account_id VARCHAR NOT NULL
FOREIGN KEY (account_id) REFERENCES account (id)

lachlanglen · 2024-03-21T17:59:41Z

schema.sql

+    deployed_at TIMESTAMP NOT NULL,
+    source_metadata VARCHAR NOT NULL,
+    owner_id VARCHAR NOT NULL,
+    chef_id VARCHAR NULL,


for consistency and clarity's sake, either remove this NULL and other instances (as it is the default), or use it consistently across all tables for any nullable columns

my personal preference would be to remove NULL as I think it's easier to read

lachlanglen · 2024-03-21T18:07:02Z

schema.sql

+    pot_id INT NOT NULL,
+    applicant_id VARCHAR NOT NULL,
+    message TEXT,
+    current_status ENUM('Pending', 'Approved', 'Rejected', 'InReview') NOT NULL,


let's actually change this to just status

lachlanglen · 2024-03-21T18:07:43Z

schema.sql

+    current_status ENUM('Pending', 'Approved', 'Rejected', 'InReview') NOT NULL,
+    submitted_at TIMESTAMP NOT NULL,
+    last_updated_at TIMESTAMP,
+    tx_hash VARCHAR,


add NOT NULL

lachlanglen · 2024-03-21T18:07:54Z

schema.sql

+    updated_at TIMESTAMP,
+    registrant_notes TEXT,
+    admin_notes TEXT,
+    tx_hash VARCHAR,


add NOT NULL

lachlanglen · 2024-03-21T18:08:53Z

schema.sql

+    notes TEXT,
+    status ENUM('Pending', 'Approved', 'Rejected', 'InReview') NOT NULL,
+    reviewed_at TIMESTAMP NOT NULL,
+    tx_hash VARCHAR,


add NOT NULL

lachlanglen · 2024-03-21T18:10:25Z

schema.sql

+    id SERIAL PRIMARY KEY,
+    recipient_id VARCHAR NOT NULL,
+    amount VARCHAR NOT NULL,
+    amount_usd DECIMAL(10, 2),


NB: this should be set when the payout is actually paid out (so it is the USD amount at the time of the payout)

so make field name more descriptive of the instance?

No changes needed, was just making a note

lachlanglen · 2024-03-21T18:10:51Z

schema.sql

+    paid_at TIMESTAMP,
+    tx_hash VARCHAR,
+    FOREIGN KEY (recipient_id) REFERENCES account (id)
+  );


add:

FOREIGN KEY (ft_id) REFERENCES account (id)

lachlanglen · 2024-03-21T18:12:37Z

schema.sql

+    amount_usd DECIMAL(10, 2),
+    ft_id VARCHAR NOT NULL,
+    paid_at TIMESTAMP,
+    tx_hash VARCHAR,


add NOT NULL

not actually sure that we need this, it's a little confusing as it could be the hash when the payout is set, or the hash when it is paid out

lachlanglen · 2024-03-21T18:13:56Z

schema.sql

+    created_at TIMESTAMP NOT NULL,
+    message TEXT,
+    resolved BOOL NOT NULL,
+    tx_hash VARCHAR,


add NOT NULL

lachlanglen · 2024-03-21T18:14:00Z

schema.sql

+    pot_id INT NOT NULL,
+    created_at TIMESTAMP NOT NULL,
+    message TEXT NOT NULL,
+    tx_hash VARCHAR,


add NOT NULL

lachlanglen · 2024-03-21T18:15:06Z

schema.sql

+-- Table donation
+CREATE TABLE
+  donation (
+    id SERIAL PRIMARY KEY,


change to INT instead of SERIAL so we can set manually and make sure it stays in sync with Donation ID on contract

lachlanglen · 2024-03-21T18:15:40Z

schema.sql

+    total_amount_usd DECIMAL(10, 2),
+    net_amount VARCHAR NOT NULL,
+    net_amount_usd DECIMAL(10, 2),
+    ft_id VARCHAR,


add NOT NULL

lachlanglen · 2024-03-21T18:15:59Z

schema.sql

+    FOREIGN KEY (donor_id) REFERENCES account (id),
+    FOREIGN KEY (pot_id) REFERENCES pot (id),
+    FOREIGN KEY (recipient_id) REFERENCES account (id)
+  );


add:

FOREIGN KEY (ft_id) REFERENCES account (id)
FOREIGN KEY (referrer_id) REFERENCES account (id)
FOREIGN KEY (chef_id) REFERENCES account (id)

lachlanglen · 2024-03-21T18:17:33Z

schema.sql

+    chef_id VARCHAR,
+    chef_fee VARCHAR,
+    chef_fee_usd DECIMAL(10, 2),
+    tx_hash VARCHAR,


add NOT NULL

lachlanglen · 2024-03-21T18:19:00Z

schema.sql

+CREATE TABLE
+  pot_admin (
+    pot_id INT NOT NULL,
+    admin_id VARCHAR NOT NULL,
+    PRIMARY KEY (pot_id, admin_id),
+    FOREIGN KEY (pot_id) REFERENCES pot (id),
+    FOREIGN KEY (admin_id) REFERENCES account (id)
+  );


move this up so it's under pot table definition

i don't understand, why should this be moved to pots?, it's more like a many to many junction table, since an account can be admin on different pots, and pots can have many admins.

No no I was just saying, all the pot-related tables should be grouped together for ease of reading & understanding the schema file

lachlanglen · 2024-03-21T18:20:07Z

schema.sql

+        'Add_List_Admin',
+        'Remove_List_Admin'
+      ) NOT NULL
+  );


add:

FOREIGN KEY (signer_id) REFERENCES account (id)
FOREIGN KEY (receiver_id) REFERENCES account (id)

lachlanglen · 2024-03-21T18:20:14Z

schema.sql

+    receiver_id VARCHAR NOT NULL,
+    timestamp TIMESTAMP NOT NULL,
+    action_result VARCHAR,
+    tx_hash VARCHAR,


add NOT NULL

lachlanglen · 2024-03-21T18:57:28Z

schema.sql

+      'Blacklisted'
+    ) NOT NULL,
+    FOREIGN KEY (owner_id) REFERENCES account (id)
+  );


add:

created_at TIMESTAMP NOT NULL updated_at TIMESTAMP

lachlanglen · 2024-03-21T19:04:19Z

schema.sql

+
+-- pot index
+
+-- CREATE INDEX "deploy_time_idx" ON pot (deployed_at);


add this back in

lachlanglen · 2024-03-21T19:04:56Z

schema.sql

+-- CREATE INDEX idx_pot_payout_ft_id ON pot_payout (ft_id);
+CREATE INDEX idx_pot_payout_paid_at ON pot_payout (paid_at);
+
+


add:

indexes for account (total_donations_received_usd, total_matching_pool_allocations_usd, total_donated_usd, donors_count - we will almost certainly want to query ordered results based on these columns)

indexes for list on created_at and updated_at

composite index for list registration/status: CREATE INDEX idx_list_id_status ON list_registration(list_id, status);

composite indexes for pot application & matching round start/end:

CREATE INDEX idx_application_period ON pot(application_start, application_end); CREATE INDEX idx_matching_period ON pot(matching_round_start, matching_round_end);

additional pot indexes on deployed_at, total_matching_pool, total_matching_pool_usd, total_public_donations, total_public_donations_usd, public_donations_count

composite index for pot_application on pot_id/status

pot_application_review index on reviewer_id

additional pot_payout indexes on amount and amount_usd

pot_payout_challenge indexes on pot_id and challenger_id, plus composite index for pot_id + created_at

pot_payout_challenge_admin_response indexes on admin_id and challenge_id

additional donation indexes on total_amount, total_amount_usd, matching_pool, recipient_id, referrer_id & chef_id

additional activity indexes on signer_id, recipient_id and type

lachlanglen · 2024-03-21T19:16:06Z

schema.sql

+CREATE INDEX idx_pot_application_start ON pot (application_start);
+CREATE INDEX idx_pot_application_end ON pot (application_end);


remove these and replace with composite indexes as per example below

lachlanglen · 2024-03-21T21:28:21Z

schema.sql

+
+-- CREATE INDEX idx_pot_payout_recipient_id ON pot_payout (recipient_id);
+-- CREATE INDEX idx_pot_payout_ft_id ON pot_payout (ft_id);
+CREATE INDEX idx_pot_payout_paid_at ON pot_payout (paid_at);


move this up with the other pot_payout indexes

lachlanglen · 2024-03-22T14:09:57Z

indexer.ts

+
+        }
+      } else {
+        potUpdate["public_donations_count"] = pot.public_donations_count || 0 + 1


lachlanglen · 2024-03-22T14:11:31Z

indexer.ts

+          ? "Donate_Pot_Matching_Pool"
+            : "Donate_Pot_Public",
+      action_result: recipient,
+      tx_hash: receiptId,


change from receipt hash to tx hash

lachlanglen · 2024-03-22T14:12:48Z

indexer.ts

+          : donation.matching_pool
+          ? "Donate_Pot_Matching_Pool"
+            : "Donate_Pot_Public",
+      action_result: recipient,


not sure about this. what do you think about potentially storing the entire raw result in this field? that way the client can get the success value, or the receiver, or whatever else they may want

we could do that, but not all actions actually have a result value, i think. but i'll change it to that

lachlanglen · 2024-03-22T17:00:14Z

indexer.ts

+              break;
+
+            // this is a form of donation where user calls donate on a pot
+            case "handle_protocol_fee_callback":


this should probably listen to the process_donation callback which is at the end of the callback chain and returns the new donation

lachlanglen · 2024-03-22T17:00:29Z

indexer.ts

+                args,
+                action.receiverId,
+                action.signerId,
+                "handle_protocol_fee_callback",


maybe change to "pot" for clarity

lachlanglen · 2024-03-22T17:00:38Z

indexer.ts

+                args,
+                action.receiverId,
+                action.signerId,
+                "donate",


maybe change to "direct" for clarity

lachlanglen · 2024-03-22T17:01:27Z

indexer.ts

+    };
+    await context.db.Donation.insert(donation);
+
+    if (actionName != "donate") { // update pot data such as public donations and matchingpool


change to actionName == "pot" for clarity?

lachlanglen · 2024-03-22T17:04:25Z

indexer.ts

+      registrant_id: data._project_id || signerId,
+      status: registry.default_registration_status,
+      submitted_at: new Date(
+        Number(BigInt(block.header().timestampNanosec) / BigInt(1000000))
+      ),


could get these from function result (ProjectExternal) if desired

yes, but except there is a chance these data source might be incorrect, i wanted to use them since they're readily available as decoding function results take extra steps including looping through receipts.

the thing is also that if it is registered by admin, it will be approved by default (at least that is the current functionality). i think it's probably better to take it from the decoded SuccessValue

Also as I noted elsewhere this morning, since the new Lists contract will be up very soon (ideally by EOD today) and existing Potlock registrations will be migrated to this contract via its register method, let's not worry about indexing the legacy Registry contract

lachlanglen · 2024-03-22T17:10:37Z

indexer.ts

+      timestamp: reg.submitted_at,
+      type: "Register",
+      action_result: reg.registrant_id,
+      tx_hash: receiptId,


use tx hash instead of receipt hash

lachlanglen · 2024-03-22T17:12:19Z

indexer.ts

+    let data = JSON.parse(args);
+    console.log("new Project data::", { ...data }, receiverId);
+    let registry = (await context.db.List.select({ id: receiverId }))[0]; // fetch the registry so as to get default
+    let reg = {


reg also needs a list_id (registry.id)

lachlanglen · 2024-03-22T17:12:43Z

schema.sql

+  );
+
+CREATE TABLE
+  list_registration (


also needs list_id

lachlanglen · 2024-03-22T17:50:49Z

indexer.ts

+    receiptId
+  ) {
+    let data = JSON.parse(args);
+    console.log("new factory data::", { ...data }, receiverId);


might want to update this log text

lachlanglen · 2024-03-22T17:51:24Z

indexer.ts

+      reviewer_id: signerId,
+      notes: update_data.notes,
+      status: update_data.status,
+      reviewed_at: update_data.updated_at,


updated_at won't be present on update_data

ermm... it won't? the chef_set_application_status function returns an application object with those values... including updated at

it appears that you are using the function call args, not the result

i was using the result, let update_data = JSON.parse(atob(receipt.status["SuccessValue"]));

oooooops sorry missed that

lachlanglen · 2024-03-22T17:51:36Z

indexer.ts

+    };
+    let applicationUpdate = {
+      current_status: update_data.status,
+      last_updated_at: update_data.updated_at,


ditto..... 😉

........ditto 😶‍🌫️

........ ditto!

lachlanglen · 2024-03-22T17:53:45Z

indexer.ts

+      default_registration_status: data.status,
+    };
+
+    await context.db.List.update({ id: receiverId }, listUpdate);


this works for the current registry contract but not for Lists contract, where the list_id will be specified in the call. changing to id: data.list_id || receiverId should resolve

lachlanglen · 2024-03-22T17:56:01Z

indexer.ts

+    console.log("set payout data::", { ...data }, receiverId);
+    let payouts = data.payouts;
+    for (const payout in payouts) {
+      // general question: should we register projects as accounts?


lachlanglen · 2024-03-22T17:56:18Z

indexer.ts

+      let potPayout = {
+        recipient_id: payout["project_id"],
+        amount: payout["amount"],
+        ft_id: payout["ft_id"] || null,


change to payout["ft_id"] || "near"

lachlanglen · 2024-03-22T18:01:06Z

indexer.ts

+        recipient_id: payout["project_id"],
+        amount: payout["amount"],
+        ft_id: payout["ft_id"] || null,
+        tx_hash: receiptId,


use tx hash instead of receipt hash

lachlanglen · 2024-03-22T18:02:03Z

indexer.ts

+    let created_at = new Date(
+      Number(BigInt(block.header().timestampNanosec) / BigInt(1000000)) // convert to ms then date
+    )


this calculation is done a lot, perhaps worth doing once for the block further up the chain and passing down?

lachlanglen · 2024-03-22T18:02:37Z

indexer.ts

+      pot_id: receiverId,
+      created_at,
+      message: data.reason,
+      tx_hash: receiptId,


use tx hash instead of receipt hash

lachlanglen · 2024-03-22T18:03:52Z

indexer.ts

+      receiver_id: receiverId,
+      timestamp: created_at,
+      type: "Challenge_Payout",
+      action_result: receiverId,


not sure this is useful

lachlanglen · 2024-03-22T18:04:02Z

indexer.ts

+      timestamp: created_at,
+      type: "Challenge_Payout",
+      action_result: receiverId,
+      tx_hash: receiptId,


use tx hash instead of receipt hash

lachlanglen · 2024-03-22T18:06:09Z

indexer.ts

+    let payout = {
+      recipient_id: data.project_id,
+      amount: data.amount,
+      paid_at: data.paid_at,
+      tx_hash: receiptId,
+    };


i don't the args (data) will be in this format you're expecting here. the transfer_payout_callback method receives a payout argument so you may have to inspect this to get project_id, amount etc

the data is the payout argument....

but won't it be at data.payout?

lachlanglen · 2024-03-22T18:06:25Z

indexer.ts

+      recipient_id: data.project_id,
+      amount: data.amount,
+      paid_at: data.paid_at,
+      tx_hash: receiptId,


change to tx hash instead of receipt hash

lachlanglen · 2024-03-22T18:07:14Z

indexer.ts

+    }
+  }
+
+  async function handlePayout(args, receiverId, signerId, receiptId) {


when is pot.all_paid_out updated?

lachlanglen · 2024-03-22T18:07:37Z

indexer.ts

+              break;
+            case "owner_remove_admins":
+              console.log(
+                "setting payot....:",


incorrect log

lachlanglen · 2024-03-22T18:11:04Z

indexer.ts

+                action.receiptId
+              );
+              break;
+            case "owner_remove_admins":


FYI there is a method with this name on all the contracts, so will need to be more specific to handle this based on the contract (registry/lists, pot factory, pot, donation)

lachlanglen · 2024-03-22T18:12:25Z

indexer.ts

+    let ts = new Date(
+      Number(BigInt(block.header().timestampNanosec) / BigInt(1000000)) // convert to ms then date
+    )
+    let list = await context.db.List.select({id: receiverId})


change to id: data.list_id || receiverId to handle lists contract also

lachlanglen · 2024-03-22T18:14:21Z

indexer.ts

+      owner_id: data.owner,
+      default_registration_status: "Approved", // the first registry contract has approved as default, and the later changed through the admin set function call, which we also listen to, so it should self correct.
+      name: receiverId.split(".")[0],
+      tx_hash: receiptId,


use tx hash not receipt hash

Prometheo added 2 commits March 21, 2024 11:25

initial indexer code

e1fcb82

remove admins

9a2c268