Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to set destination while using uploadManyFiles in TransferManager #2492

Closed
arnav-kr opened this issue Jun 26, 2024 · 4 comments · Fixed by #2497
Closed

How to set destination while using uploadManyFiles in TransferManager #2492

arnav-kr opened this issue Jun 26, 2024 · 4 comments · Fixed by #2497
Assignees
@ddelgrosso1
Labels
api: storage Issues related to the googleapis/nodejs-storage API. priority: p3 Desirable enhancement or fix. May not be included in next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. type: question Request for information or clarification. Not an issue.

Comments

@arnav-kr
Copy link

arnav-kr commented Jun 26, 2024 •
edited
Loading

I have a local directory and I want to upload its contents to cloud storage, but when using transferManager.uploadManyFiles with either the directory name, or array of files, the destination of the file in cloud storage is taken from the local file path.

local directory structure:

.
└── data
    └── avatars
        └── <image files>

here's my code

import { Storage, TransferManager } from "@google-cloud/storage";

const storage = new Storage();
const transferManager = new TransferManager(storage.bucket("bucket-name"));

const res = await transferManager.uploadManyFiles("data/avatars", {
  prefix: "avatars",
  passthroughOptions: {
    destination: "", // ???
  },
  skipIfExists: true,
});

when uploaded with following code, it becomes

.
└── avatars
    └── data
        └── avatars
            └── <image files>

the prefix property works, but I want to upload files directly to avatars/
there seem to be a passthroughOptions that can take a destination, but when using uploadManyFiles we don't know the exact file names, so can't make the full path

Shouldn't it be that while using prefix, property, the local file path shouldn't be considered?
@arnav-kr arnav-kr added priority: p3 Desirable enhancement or fix. May not be included in next release. type: question Request for information or clarification. Not an issue. labels Jun 26, 2024
@product-auto-label product-auto-label bot added the api: storage Issues related to the googleapis/nodejs-storage API. label Jun 26, 2024
@arnav-kr
Copy link
Author

For a temporary fix, I edited library source to ignore local paths when prefix is supplied, it works, I'd still like to know the intended way of doing so

@ddelgrosso1 ddelgrosso1 self-assigned this Jun 26, 2024
@ddelgrosso1
Copy link
Contributor

Hi @arnav-kr thanks for the question. The original intent was to mirror the same directory structure from local to storage. Obviously as you bumped into this can be improved. Currently destination isn't directly settable in uploadMany

nodejs-storage/src/transfer-manager.ts

Lines 95 to 100 in 18eef67

export interface UploadManyFilesOptions {
concurrencyLimit?: number;
skipIfExists?: boolean;
prefix?: string;
passthroughOptions?: Omit<UploadOptions, 'destination'>;
}
.

We can probably add some additional options to offer a greater degree of flexibility such as not mirroring the entire path structure and only using prefix. I can take a look at making these improvements in the near future.

@arnav-kr
Copy link
Author

Probably in options, something like a destinationBuilder would make it more flexible, than specifing a string, as this way we can get the filename and process accodingly

...
destinationBuilder: (path, additionalParams) => {
  let dest;
  // perform operation on path
  return dest;
}
...

@ddelgrosso1 ddelgrosso1 added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. type: question Request for information or clarification. Not an issue. and removed type: question Request for information or clarification. Not an issue. labels Jul 9, 2024
@ddelgrosso1 ddelgrosso1 mentioned this issue Jul 9, 2024
Merged
4 tasks
@arnav-kr
Copy link
Author

Thanks ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ddelgrosso1 ddelgrosso1

Labels
api: storage Issues related to the googleapis/nodejs-storage API. priority: p3 Desirable enhancement or fix. May not be included in next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. type: question Request for information or clarification. Not an issue.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: add function to allow user to set destination in transfer manager ddelgrosso1/nodejs-storage
2 participants
@arnav-kr @ddelgrosso1