Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: read time should be used for transaction reads #1171

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
b66827e
Allow datastore projectId to be fetched from clien
danieljbruce Sep 7, 2023
684c469
Create a file for mocking out commits
danieljbruce Sep 20, 2023
1ec6ed2
Create a test to measure latency of call.
danieljbruce Sep 21, 2023
40213ca
Run the linting fixes
danieljbruce Sep 21, 2023
57e3b13
Add license header to top of test file
danieljbruce Sep 21, 2023
af56d91
Add comment for test for now
danieljbruce Sep 21, 2023
72f9492
Add a test for the mock server
danieljbruce Sep 22, 2023
194e109
Set current retry attempt to 0
danieljbruce Sep 22, 2023
22d5f54
Add a log for call time
danieljbruce Sep 22, 2023
81f5956
Add another mock
danieljbruce Sep 25, 2023
dc9dbb7
Eliminate code from the mock file
danieljbruce Oct 13, 2023
0ad0a57
Start off by adding read time to read options
danieljbruce Oct 13, 2023
e795c3a
Update the test to use transactions
danieljbruce Oct 13, 2023
851877c
Remove only
danieljbruce Oct 13, 2023
b9bfd79
Remove the before hook
danieljbruce Oct 13, 2023
617b013
Remove unnecessary cherry picked files
danieljbruce Oct 13, 2023
4a32ad3
Clean up PR diff
danieljbruce Oct 13, 2023
8f74353
clean up PR diff
danieljbruce Oct 13, 2023
4c9445c
Update the test so that it is run as a transaction
danieljbruce Oct 18, 2023
32729e1
Add an integration test
danieljbruce Oct 18, 2023
6ef2800
Merge branch 'main' of https://github.com/googleapis/nodejs-datastore…
danieljbruce Oct 30, 2023
1354b47
Linting fixing indents
danieljbruce Oct 30, 2023
80ca7ec
Merge branch 'main' into not-use-read-time-for-run-tx
danieljbruce Apr 4, 2024
136f1b7
Update the header
danieljbruce Apr 4, 2024
280244d
Merge branch 'not-use-read-time-for-run-tx' of https://github.com/dan…
danieljbruce Apr 4, 2024
95d0d54
Fix unit test
danieljbruce Apr 4, 2024
aa4f125
System test changes.
danieljbruce Apr 4, 2024
b78b091
Modify test
danieljbruce Apr 4, 2024
cc72220
Replace with less precise assert
danieljbruce Apr 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions src/request.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
KeyProto,
ResponseResult,
Entities,
ValueProto,

Check warning on line 47 in src/request.ts

View workflow job for this annotation

GitHub Actions / lint

'ValueProto' is defined but never used
} from './entity';
import {
Query,
Expand Down Expand Up @@ -609,9 +609,9 @@
const results = res.batch.aggregationResults;
const finalResults = results
.map(
(aggregationResult: any) => aggregationResult.aggregateProperties

Check warning on line 612 in src/request.ts

View workflow job for this annotation

GitHub Actions / lint

Unexpected any. Specify a different type
)
.map((aggregateProperties: any) =>

Check warning on line 614 in src/request.ts

View workflow job for this annotation

GitHub Actions / lint

Unexpected any. Specify a different type
Object.fromEntries(
new Map(
Object.keys(aggregateProperties).map(key => [
Expand Down Expand Up @@ -946,7 +946,7 @@
callback?: SaveCallback
): void | Promise<CommitResponse> {
const transaction = this.datastore.transaction();
transaction.run(async (err: any) => {

Check warning on line 949 in src/request.ts

View workflow job for this annotation

GitHub Actions / lint

Unexpected any. Specify a different type
if (err) {
try {
await transaction.rollback();
Expand Down Expand Up @@ -1025,9 +1025,13 @@
);
}

reqOpts.readOptions = {
transaction: this.id,
};
if (reqOpts.readOptions) {
Object.assign(reqOpts.readOptions, {transaction: this.id});
} else {
reqOpts.readOptions = {
transaction: this.id,
};
}
}

datastore.auth.getProjectId((err, projectId) => {
Expand Down
42 changes: 42 additions & 0 deletions system-test/datastore.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1931,6 +1931,29 @@ async.each(
});
});
describe('transactions', () => {
before(async () => {
// This 'sleep' function is used to ensure that when data is saved to datastore,
// the time on the server is far enough ahead to be sure to be later than timeBeforeDataCreation
// so that when we read at timeBeforeDataCreation we get a snapshot of data before the save.
const key = datastore.key(['Company', 'Google']);
function sleep(ms: number) {
return new Promise(resolve => setTimeout(resolve, ms));
}
// Save for a key so that a read time can be accessed for snapshot reads.
const emptyData = {
key,
data: {},
};
await datastore.save(emptyData);
// Sleep for 10 seconds to ensure timeBeforeDataCreation includes the empty data
await sleep(10000);
timeBeforeDataCreation = await getReadTime([
{kind: 'Company', name: 'Google'},
]);
// Sleep for 10 seconds so that any future reads will be later than timeBeforeDataCreation.
await sleep(10000);
});

it('should run in a transaction', async () => {
const key = datastore.key(['Company', 'Google']);
const obj = {
Expand Down Expand Up @@ -2031,6 +2054,25 @@ async.each(
await transaction.commit();
});

it('should query within a transaction at a previous read time', async () => {
const transaction = datastore.transaction();
await transaction.run();
const query = transaction.createQuery('Company');
let entitiesBefore;
let entitiesNow;
try {
[entitiesBefore] = await query.run({
readTime: timeBeforeDataCreation,
});
[entitiesNow] = await query.run({});
} catch (e) {
await transaction.rollback();
return;
}
assert(entitiesBefore!.length < entitiesNow!.length);
await transaction.commit();
});

describe('aggregate query within a transaction', async () => {
it('should run a query and return the results', async () => {
// Add a test here to verify what the data is at this time.
Expand Down
118 changes: 118 additions & 0 deletions test/gapic-mocks/runQuery.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
// Copyright 2024 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

import * as assert from 'assert';
import {describe} from 'mocha';
import {DatastoreClient, Datastore} from '../../src';
import * as protos from '../../protos/protos';
import {Callback} from 'google-gax';

describe('Run Query', () => {
const PROJECT_ID = 'project-id';
const NAMESPACE = 'namespace';
const clientName = 'DatastoreClient';
const options = {
projectId: PROJECT_ID,
namespace: NAMESPACE,
};
const datastore = new Datastore(options);
// By default, datastore.clients_ is an empty map.
// To mock out commit we need the map to contain the Gapic data client.
// Normally a call to the data client through the datastore object would initialize it.
// We don't want to make this call because it would make a grpc request.
// So we just add the data client to the map.
const gapic = Object.freeze({
v1: require('../../src/v1'),
});
datastore.clients_.set(clientName, new gapic.v1[clientName](options));

// This function is used for doing assertion checks.
// The idea is to check that the right request gets passed to the commit function in the Gapic layer.
function setRunQueryComparison(
compareFn: (request: protos.google.datastore.v1.IRunQueryRequest) => void
) {
const dataClient = datastore.clients_.get(clientName);
if (dataClient) {
dataClient.runQuery = (
request: any,
options: any,
callback: (
err?: unknown,
res?: protos.google.datastore.v1.IRunQueryResponse
) => void
) => {
try {
compareFn(request);
} catch (e) {
callback(e);
}
callback(null, {
batch: {
moreResults:
protos.google.datastore.v1.QueryResultBatch.MoreResultsType
.NO_MORE_RESULTS,
},
});
};
}
}

it('should pass read time into runQuery for transactions', async () => {
// First mock out beginTransaction
const dataClient = datastore.clients_.get(clientName);
const testId = Buffer.from(Array.from(Array(100).keys()));
if (dataClient) {
dataClient.beginTransaction = (
request: protos.google.datastore.v1.IBeginTransactionRequest,
options: protos.google.datastore.v1.IBeginTransactionResponse,
callback: Callback<
protos.google.datastore.v1.IBeginTransactionResponse,
| protos.google.datastore.v1.IBeginTransactionRequest
| null
| undefined,
{} | null | undefined
>
) => {
callback(null, {
transaction: testId,
});
};
}
setRunQueryComparison(
(request: protos.google.datastore.v1.IRunQueryRequest) => {
assert.deepStrictEqual(request, {
readOptions: {
transaction: testId,
readTime: {
seconds: 77,
},
},
partitionId: {
namespaceId: 'namespace',
},
query: {
distinctOn: [],
kind: [{name: 'Task'}],
order: [],
projection: [],
},
projectId: 'project-id',
});
}
);
const transaction = datastore.transaction();
const query = datastore.createQuery('Task');
await transaction.runQuery(query, {readTime: 77000});
});
});
Loading